Understanding politics with Watson using Text Analytics

To understand the topics that actually are important to different political parties is a difficult task. Can text analytics together with an search index be an approach to given a better understanding?

This blog post describes how IBM Watson Explorer Content Analytics (WCA) can be used to make sense of Swedish politics. All speeches (in Swedish: anföranden) in the Swedish Parliament from 2004 to 2015 are analyzed using WCA. In total 139 110 transcribed text documents were analyzed. The Swedish language support build by Findwise for WCA is used together with a few text analytic processing steps which parses out person names, political party, dates and topics of interest. The selected topics in this analyzed are all related to infrastructure and different types of fuels.

We start by looking at how some of the topics are mentioned over time.

Analyze of terms of interets in Swedsih parlament between 2004 and 2014.

Analyze of terms of interest in Swedish parliament between 2004 and 2014.

The view shows topic which has a higher number of mentions compared to what would be expected during one year. Here we can see among other topics that the topic flygplats (airport) has a high increase in number of mentioning during 2014.

So let’s dive down and see what is being said about the topic flygplats during 2014.

Swedish political parties mentioning Bromma Airport.

Swedish political parties mentioning Bromma Airport during 2014.

The above image shows how the different political parties are mentioning the topic flygplats during the year 2014. The blue bar shows the number of times the topic flygplats was mentioned by each political party during the year. The green bar shows the WCA correlation value which indicates how strongly related a term is to the current filter. What we can conclude is that party Moderaterna mentioned flygplats during 2014 more frequently than other parties.

Reviewing the most correlated nouns when filtering on flygplats and the year 2014 shows among some other nouns: Bromma (place in Sweden), airport and nedläggning (closing). This gives some idea what was discussed during the period. By filtering on the speeches which was held by Moderaterna and reading some of them makes it clear that Moderaterna is against a closing of Bromma airport.

The text analytics and the index provided by WCA helps us both discover trending topics over time and gives us a tool for understanding who talked about a subject and what was said.

All the different topics about infrastructure can together create a single topic for infrastructure. Speeches that are mentioning tåg (train), bredband (broadband) or any other defined term for infrastructure are also tagged with the topic infrastructure. This wider concept of infrastructure can of course also be viewed over time.

Discussions in Swedish parliament mentioning the defined terms which builds up the subject infrastructure 2004 to 2015.

Discussions in Swedish parliament mentioning the defined terms which builds up the subject infrastructure 2004 to 2015.

Another way of finding which party that are most correlated to a subject is by comparing pair of facets. The following table shows parties highly related to terms regarding infrastructure and type of fuels.

Political parties highly correlated to subjects regarding infrastructure and types of fuel.

Swedish political parties highly correlated to subjects regarding infrastructure and types of fuel.

Let’s start by explain the first row in order to understand the table. Mobilnät (mobile net) has only been mentioned 44 times by Centerpartiet, but Centerpartiet is still highly related to the term with a WCA correlation value of 3.7. This means that Centerpartiet has a higher share of its speeches mentioning mobilnät compared to other parties. The table indicates that two parties Centerpartiet and Miljöpartiet are more involved about the subject infrastructure topics than other political parties.

Swedish parties mentioning the defined concept of infrastructure.

Swedish parties mentioning the defined concept of infrastructure.

Filtering on the concept infrastructure also shows that Miljöpartiet and Centerpartiet are the two parties which has the highest share of speeches mentioning the defined infrastructure topics.

Interested to dig deeper into the data? Parsing written text with text analytics is a successful approach for increasing an understanding of subjects such as politics. Using IBM Watson Explorer Content Analytics makes it easy. Most of the functionality used in this example is also out of the box functionalities in WCA.

Swedish language support (natural language processing) for IBM Content Analytics (ICA)

Findwise has now extended the NLP (natural language processing) in ICA to include both support for Swedish PoS tagging and Swedish sentiment analysis.

IBM Content Analytics with Enterprise Search (ICA) has its strength in natural language processing (NLP) which is achieved in the UIMA pipeline. From a Swedish perspective, one concern with ICA has always been its lack of NLP for Swedish. Previously the Swedish support in ICA consisted only of dictionary-based lemmatization (word: “sprang” -> lemma: “springa”). However, for a number of other languages ICA has also provided part of speech (PoS) tagging and sentiment analysis. One of the benefits of the PoS tagger is its ability to disambiguate words, which belong to multiple classes (e.g. “run” can be both a noun and a verb) as well as assign tags to words, which are not found in the dictionary. Furthermore, the POS tagger is crucial when it comes to improving entity extraction, which is important when a deeper understanding of the indexed text is needed.

Findwise has now extended the NLP in ICA to include both support for Swedish PoS tagging and Swedish sentiment analysis. The two images below shows simple examples of the PoS support.

Example when ICA uses NLP to analyse the string "ICA är en produkt som klarar entitetsextrahering"Example when ICA uses NLP to analyse the string "Watson deltog i jeopardy"

The question is how this extended functionality could be used?

IBM uses ICA and its NLP support together with several of their products. The jeopardy playing computer Watson may be the most famous example, even if it is not a real product. Watson used NLP in its UIMA pipeline when it analyzed its data from sources such as Wikipedia and Imdb.

One product which leverage from ICA and its NLP capabilities is Content and Predictive Analytics for Healthcare. This product helps doctors to determine which action to take for a patient given the patient’s journal and the symptoms. By also leveraging the predictive analytics from SPSS it is possible to suggest the next action for the patient.

ICA can also be connected directly to IBM Cognos or SPSS where ICA is the tool which creates structure to unstructured data. By using the NLP or sentiment analytics in ICA, structured data can be extracted from text documents. This data can then be fed to IBM Cognos, SPSS or non IBM products such as Splunk.

ICA can also be used on its own as a text miner or a search platform, but in many cases ICA delivers its maximum value together with other products. ICA is a product which helps enriching data by creating structure to unstructured data. The processed data can then be used by other products which normally work with structured data.

Analytics and Big Data at IBM Information On Demand 2011

The big trend these days are in Big Data and how you can analyze large amounts of information in order to gain important insights, and from those insights be able to take the right action. This trend was a hot topic at the IBM Information On Demand (IOD) conference in Las Vegas earlier this year. IBM has a very strong position in this field, it’s hard to have missed how their computer system Watson challenged the top players of all time in Jeopardy recently, and won! Read more about Watson

Now IBM has taken the technology behind Watson and started to apply it in their different analytics products, where one specific area that is being targeted is healthcare. For this area IBM released a new product during IOD called IBM Content and Predictive Analytics for Healthcare, which can for example be used as a tool for physicians to support them in their diagnosis of patients.

In April this year IBM merged two of their products, their search engine OmniFind and their product for analyzing large amounts of unstructured information, Content Analytics. The new product is called IBM Content analytics with Enterprise search and it too is based on much of the same technology that is used in Watson, more specifically it utilizes the same Natural Language Processing techniques. This means that it has the ability to understand text on a level just as sophisticated as that of Watson.

Content Analytics with enterprise search scales very well to many millions of documents. However, when there is a need for analyzing really enormous data sets, in the magnitude of petabytes or even exabytes, IBM has developed what they call their BigData platform. This platform mainly revolves around two products, InfoSphere Streams and InfoSphere BigInsights, and it builds on a foundation of open source software, such as Apache Hadoop and Apache Lucene. InfoSphere Streams is used for real time analysis of information in motion. This helps you understand what’s happening right at this moment in your organization and supports you in taking appropriate action as things are happening. InfoSphere BigInsights on the other hand lets you analyze and draw insight from massive amounts of already existing data.

Studies have shown how organizations that fall short in this area are overtaken by those who understand how to use the power of analytics.

IBM has surely chosen an interesting path when merging Analytics with Findability.

Findability Blog: Wrapping up the 2010 posts

Christmas is finally here and at Findwise we are taking a few days off to spend time with family and friends.

During 2010 we’ve delivered more than 25 successful projects, arranged breakfast seminars to talk about customer solutions (based on Microsoft, IBM, Autonomy and Open source), meet-ups in a number of cities as well as networking meetings for profound Findability discussions and moving in parties for our new offices.

At our Findability blog we have been discussing technology and vendor solutions (Microsoft and FAST, Autonomy, IBM, Google and open source), researchconferences, customized solutions and how to find a balance between technology and people.

Some of our posts have resulted in discussions, both on our own blog and in other forums. Please get involved in some of the previous ongoing discussions on “Solr Processing Pipeline”,  “Search and Business Intelligence” or “If a piece of content is never read, does it exist?”  if you have thoughts to share.

Findability blog is taking a break and we will be back with new posts is January.

If you have some spare time during the vacation some of customers run their own blogs, and good reading tips within Findability are the blogs driven by Kristian Norling (VGR) and Alexandra Larsson (Swedish armed forces).

Merry Christmas and a Happy New Year to you all!

OmniFind Enterprise Edition 9.1 – New Capabilities Discussed Over Breakfast

During the last year a number of interesting things has happened to IBM’s search platform and the new version, OmniFind 9.1, was released this summer. Apart from a large number of improvements in the interface, the change to basing the new solution on open source (Lucene) has proven to be a genius by-pass of some of OmniFinds previous shortcomings.

The licensing model is still quite complicated, something Stephen E Arnold highlighted earlier this year. Since a number of our customers have chosen to take a closer look at OmniFind as a search solution we decided to host a breakfast seminar together with IBM last Thursday, in order to discuss the new features and show how some of our customer are working with it.

Without a doubt, the most interesting part is always to discuss how the solution can be utilized for intranets, extranets, external sites and e-business purposes.

Apart from this, we also took a look at some of the new features:
Type ahead (query suggestion), based on either search statistics or indexed content

Type ahead

Faceted search i.e. the ability to filter on dates, locations, format etc as well as numeric and date range. The later is of course widely used within e-business.

Facets for e-business

Thumbnail views of documents (yes, exactly what it sounds like: a thumbnail view for first page of documents in results page)

Thumbnail of a document

Search analytics in OmniFind 9.1 holds a number of interesting statistic capabilities. Some things worth mentioning is number of queries, query popularity, number of users, average response time (ms) and worst response time (ms).

Save searches (to be able to go back and see if new information has been included), search within result sets (to further narrow your result set within a given result set) and did-you-mean functionality (spell checking) are also included.

..and improvements on the administrator side, just to mention a few:

  • Ability to change the relevancy i.e. to adjust and give certain types of information higher ranking
  • Support for incremental indexing i.e. to only re-index the information that is new or changed since the last time you made it searchable

To conclude: IBM is making a whole lot of improvements in the new version, which are worth taking a closer look at. During the spring we are running upgrading projects for some of our customers, and we will keep you up-to-date with the different application areas OmniFind Enterprise Edition 9.1 is being used for. Please let us know if you have any particular questions or have areas that you are interested in.

Enterprise Search and Business Intelligence?

Business Intelligence (BI) and Enterprise Search is a never ending story

A number of years ago Gartner coined “Biggle” – which was an expression for BI meeting Google. Back then a number of BI vendors, among them Cognos and SAS, claimed that they were working with enterprise search strategically (e.g. became Google One-box partners). Search vendors, like FAST, Autonomy and IBM also started to cooperate with companies such as Cognos. “The Adaptive Warehouse” and “BI for the masses” soon became buzzwords that spread in the industry.

The skeptics claimed that enterprise search never would be good at numbers and that BI would never be good with text.

Since then a lot a lot has happened and today the major vendors within Enterprise Search all claim to have BI solutions that can be fully integrated (and the other way around – BI solutions that can integrate with enterprise search).

The aim is the same now as back then:  to provide unified access to both structured (database) and unstructured (content) corporate information. As FAST wrote in a number of ‘Special Focus’:

“Users should have access to a wide variety of data from just one, simple search interface, covering reports, analysis, scorecards, dashboards and other information from the BI side, along with documents, e-mail and other forms of unstructured information.”

And of course, this seems appealing to customers. But does access to all information really make us more likely to take the right decisions in terms of Business Intelligence. Gartner is in doubt.

Nigel Rayner, research vice president at Gartner Inc, says that:

”The problem isn’t that they (users) don’t have access to information or tools; they already have too much information, and that’s just in the structured BI world. Now you want to couple it with unstructured data? That’s a whole load of garbage coming from the outside world”.

But he also states that search can be used as one part of BI:

“Part of the problem with traditional BI is that it’s very focused on structured information. Search can help with getting access to the vast amount of structured information you have”

Looking at the discussions going on in forums, in blogs and in the research domain most people seem to agree with Gartner’s view: enterprise search and business intelligence makes a powerful combination, but the integrations needs to be made with a number of things in mind:

Data quality

As mentioned before, if one wants to make unstructured and structured information available as a complement to BI it needs to be of a good quality. Knowing that the information found is the latest copy and written by someone with knowledge of the area is essential. Bad information quality is a threat to an Enterprise Search solution, to a combined BI- and search solution it can be devastating. Having Content Lifecycles in place (reviewing, deleting, archiving etc) is a fundamental prerequisite.

Data analysis

Business Intelligence in traditionally built on pre-thought ideas of what data the users need, whereas search gives access to all information in an ad-hoc manner. To combine these two requires a structured way of analyzing the data. If the unstructured information is taken out of its context there is a risk that decisions are built on assumptions and not fact.

BI for the masses?

The old buzzwords are still alive, but the question mark remains. If one wants to give everyone access to BI-data it has to be clear what the purpose is. Giving people a context, for example combining the latest sales statistics with searches for information about the ongoing marketing activities serves a purpose and improves findability. Just making numbers available does not.

enterprise search and business intelligence dashboard

Business intelligence and enterprise search in a combined dashboard – vision or reality within a near future?

So, to conclude: Gartner’s vision of “Biggle” is not yet fulfilled. There are a number of interesting opportunities for the business to create findability solutions that combines business intelligence and enterprise search, but the strategies for adopting it needs to be developed in order to create the really interesting cases.

Have you come across any successful enterprise search and business intelligence integrations? What is your vision? Do you think the integration between the two is a likely scenario?

Please let us know by posting your comments.

It’s soon time for us to go on summer vacation.

If you are Swedish, Nicklas Lundblad from Google had an interesting program about search (Sommar i P1) the other day, which is available as a podcast.

Have a nice summer all of you!

Findwise releases Open Pipeline Plugins

Findwise is proud to announce that we now have released our first publicly available plugins to the Open Pipeline crawling and document processing framework. A list of all available plugins can be found on the Open Pipeline Plugins page and the ones Findwise have created can be downloaded on our Findwise Open Pipeline Plugins page.

OpenPipeline is an open source software for crawling, parsing, analyzing and routing documents. It ties together otherwise incomplete solutions for enterprise search and document processing. OpenPipeline provides a common architecture for connectors to data sources, file filters, text analyzers and modules to distribute documents across a network. It includes a job scheduler and a full UI with a point-and-click interface.

Findwise have been using this framework in a number of customer projects with great success. It ties particularly good together with Apache Solr, not only because it is open source but most importantly because it fills a hole in functionality that Solr lacks – an easy to use framework for developing document processors and connectors. However we are not using this for Solr only, a number of plugins for the Google Search Appliance have also been made and we have started investigating how Open Pipeline can be integrated with the IBM Omnifind search engine as well.

The best thing with this framework is that it is very flexible and customizable but still easy to use AND, maybe most importantly for me as a developer, easy to work with and develop against. It has a simple yet powerful enough API to handle all that you need. And because it is an open source framework any shortcomings and limitations that we find along the way can be investigated in detail and a better solution can be proposed to the Open Pipeline team for inclusion in future releases.

We have in fact already contributed to the development of the project in a great deal by using it, testing it and by reporting bugs and suggested improvements on their forums. And the response from the team has been very good – some of our suggested improvements have already been included and some are on the way in the new 0.8 version. We are also in the process of further deepening the collaboration by signing a contributors agreement so that we eventually can be able to contribute with code as well.

So how do our customers benefit from this?

First it makes us develop and deliver search and index solutions more quickly and of better quality to our customers. This is because more developers can work with the same framework as a base and the overall code base will be used more, tested more and is thus of better quality. We have also the possibility to reuse good and well tested components so that several customers together can share the costs of development and thus get a better service/product for less money which is always a good thing of course!

Autonomy Extends its Business Model with Compliance

On January 22:nd, Interwoven entered into a definitive agreement to be acquired by Autonomy, for a total transaction value of approximately $775 million.

Ever since Microsoft’s acquisition of FAST there have been quite a few discussions in blogs and forums whether any of the giants such as Oracle, IBM or even Google would make an attractive offer for Autonomy. The company has, during the last few years, always been a clear leading candidate within Enterprise Search and its range of offerings within Multimedia, Digital Asset Management and Information management solutions has made the customer base grow faster than ever.

However, the acquisition of Interwoven shows that Autonomy is serious when it comes to growing on its own premises. The product portfolio now contains a strong offering for legal and compliance and Autonomy has, once more, proven that they prefer to extend and strengthen their business model.

According to the press release the companies ‘share a vision to fundamentally change the way organizations discover, analyze and manage information’.

Beyond the buzzwords, there seems to be a clear vision and it will be interesting to see how this affects Enterprise Search in general and Autonomys future direction in particular.

Basic Enterprise Search is a Commodity – Let’s Go Further! Trends for 2008

So looking ahead, what are the trends for enterprise search 2008? Well, we have already been talking about Microsoft and IBM and there are a few other vendors (such as Google and Oracle) that have presented ways to develop their enterprise search solutions (looking at clustering, categorisation, taxonomies, entity extraction, visualisation etc.). To conclude: the simple search box is soon a commodity – a search solution for enterprises has to go beyond this.

As a result, experts believe that 2008 will become a year with stronger cooperation and more strategic differentiation among the leading vendors.

BI and search is one example, search within Rich Media (video, speech and music) another. The largest vendors are looking at new ways to develop and enrich their search capabilities and acquisitions of, or cooperation with, other companies is most likely to continue.

The next trend for 2008 is something that I personally look forward to: the user revolution
Enterprise search has for a long time been focusing on technical aspects, such as ability to scale, linguistics features etc. The predicted new era uses enterprise search as a tool based on user needs. At Findwise we have rapidly become clear of this since the need for our usability expert is great. Her work focuses on the end-user, and how the technical platforms can be used to support his or her everyday work. My believe is that this perspective will bridge the gap between IT and business, making enterprise search a tool to enable information retrieval and access in new ways.

What is your opinion? Are there more things to add for the future of enterprise search the following year?

Findwise wish you all a Happy New Year and look forward hearing from you in 2008!