Tag Archives: big data

Impressions from Findability Day 2013

Posted on by
1

We at Findwise host Findability Day to raise awareness of the importance of enterprise search and search in business, big data and to share best practices in implementation and management as well as inform about technology developments. Attending and being part of Findability Day this year was a real energy boost for all of us at Findwise. We were about 200 attendants all with focus on search and big data.

On stage, we had some very inspiring presentations. Martin White explaining the journey of search and pointing out its future direction showed how the principles of search have been around for decades. What we use it for and how we approach it is key along with enabling technology. Martin has also written a blog post about his impressions from the event, read it here.

Ravi Mynampaty of Harvard Business School showed how the search journey has evolved within Harvard Business School. One take away was the importance of step by step implementation and ensuring satisfied stakeholders along the way. Christian Finstad of Meltwater explained how they connected business values with technology to convince their clients. I think an internal decision within an organization needs similar argumentation in order to win acceptance.

Johan Johansson gave a thorough presentation about the search project at the Municipality of Norrköping. This was a tight budget project with strong deliverables. One thing to remember from Johan was his talk about “you need to try it out yourself – do the most common searches and experience it”.

DJ Skillman from Splunk, Troels Walsted Hansen from Microsoft and Daniel Bergqvist from Google gave some interesting insights into new technologies. How Splunk can be used to just harvest every imaginable data type, just bring it in and worry about using it later. How Google want to align enterprise search with consumer search and Microsofts Facebook inspired developments within graph search.

We also had some great breakout sessions with Jonas Berg, Svensk Byggtjänst who showed us their partner search application, Martin Öhléen, SKF who talked about mobility, Sebastian Forseland, Husqvarna who gave an expert lecture on master data management and Niclas Lillman&Nicklas Eriksson, Scania who talked about their journey towards a common search solution for all their knowledge workers.

If you weren’t there or if you just want to see it again we have posted videos of the presentations and most of the slides here.

We would like to thank everybody who came to the event – you made it a real success. A big thanks also goes out to our sponsors Google and Splunk who made this event possible.

The networking possibilities at the location were great and really demonstrated how the Search industry is growing.  We are very happy with the event but there is of course always room for improvement for next year. Make sure to be there!

Big data and cloud solutions at Atea Bootcamp

Posted on by
Reply

After attending the very well organized and inspiring event Atea Bootcamp 2013 I want to share some of what was said about big data and about the cloud.

Data is the new oil

On the topic of big data Atea had several speakers, one of which was Niklas Andersson, the Swedish CEO of Cisco who talked about the internet of everything. With more and more devices connected to the Internet the modern world produces massive amounts of data that, to a large extent, is unstructured and transient. It comes from a variety of sources and types – as text, video, geospatial data, information captured by a sensor in a plant or a vehicle, or from social interaction via the web. One might argue that big data is nothing new, that it is just a buzz word summarizing what has been going on for many years. However, even though we already have a perception of Big data and ideas about how to handle it and use it, we are still just scraping at the surface of what will come. According to Mr Andersson 99 percent of what could be connected to the Internet still remains to be connected. What happens when we start connecting all those things? A mind blowing perspective that makes a good case for IBMs statement that data is the new oil. For us at Findwise this is of course a highly interesting field where our knowledge can be put to good use. We recently joined Spotify among others in a big data analytics research project led by SICS. Read more about it here.

Findability in the cloud

Steve Dietch, Vice president of HP Worldwide Cloud, gave an insight into the developments of cloud services and the driving forces that control IT decision makers. According to him their customers usually have two main concerns about moving into the cloud; security and choice. Security is an obvious issue and for some organizations there are regulatory aspects to it as well. The aspect of choice has to do with the pace of development and uncertainty about which vendors will dominate the field in the future and what will become industry standards. IT departments everywhere are afraid of vendor lock in. Putting all your data in the hands of an external supplier is understandably a scary concept. What happens if you want to move it? What about how it is organized? I see an obvious case for state of the art search solutions to help handling some of these issues and to relieve some of the worries from IT departments that their data will get lost. With good findability it will not matter where it is stored or which vendor provides the cloud solution.

In conclusion, big data is big business and even though different aspects of it make for different definitions of the concept it is undeniably going to have a huge impact on all of us.

Big Data is a Big Challenge

Posted on by
Reply

Big Data is also a Big Challenge for a number of companies that would like to be ahead of the competition. I think Findwise can help a lot with both technical expertise in text analytics and search technology but also with how to put Big Data to use in a business.

During the last days of February I had the pleasure to attend IDG Big Data conference in Warsaw, Poland. It brought plenty of people from both vendors and industry that shared interesting insights on the topic. In general, big vendors that try to be associated with Big Data dominated the conference. IBM, SAS, SAP, Teradata has provided massive marketing information on software products and capabilities around Big Data. Interestingly every single presentation had its own definition on what Big Data is. This is probably caused by the fact that everybody tries to find the best definitions for fitting own products into it.

From my perspective it was very nice to hear that everyone agrees text analytics and search components are of big importance in any Big Data solution. In multiple applications analysis (both predictive and deductive) and for mass social media one must use advanced linguistic techniques for retrieving and structuring the data streams. This sounded especially strong in IBM and SAS presentations.

A couple of companies revealed what they have already achieved in so called Big Data. Orange and T-Mobile presented their approach of extending traditional business intelligence to harness Big Data. They want to go beyond standard data collected in transaction databases and open up for all the information they have from calls (picked and non-answered), SMS, data transmission logs, etc. Telecom companies consider this kind of information to be a good source for data about their clients.

But the most interesting sessions were held by companies that openly shared their experience about evolution of their Big Data solutions based mainly on open source software. In this way Adam Kawa from Spotify showed how they based their platform on Hadoop cluster starting from a single server to a few hundreds nowadays. To me that seems like a good way to grow and adapt easily to changing business needs and altering external conditions.

Nasza Klasa – a Polish Facebook competitor had a very good presentation on several dimensions connected to challenges in Big Data solutions that might be used for summarisation of this post:

  1. Lack of legal regulations – Currently there are no clear regulations on how the data might be used and how to make money out of it. It is especially important for social portals where all our personal information might be used for different kinds of analysis and sold in aggregated or non-aggregated form. But the laws might be changed soon, thus changing the business too.
  2. Big Data is a bit like research – it is hard to predict return on investment on Big Data as it is a novelty but also a very powerful tool. For many who are looking into this the challenge is internal, to convince executives to invest in something that is still rather vague.
  3. Lack of data scientists – even if there are tools for operating on Big Data, there is a huge lack of skilled people – Big Data operators. These are not IT people nor developers but rather open-minded people with a good mathematical background able to understand and find patterns in a constantly growing stream of various structured and unstructured information.

As I stated at the beginning of this post, Big Data is also a Big Challenge for a number of companies that would like to be ahead of the competition. I truly believe we at Findwise can help a lot within this area, we have both the technical expertise and experience on how to put Big Data to use in a business.

Predictive Analytics World 2012

Posted on by
Reply

At the end of November 2012 top predictive analytics experts, practitioners, authors and business thought leaders met in London at Predictive Analytics World conference. Cameral nature of the conference combined with great variety of experiences brought by over 60 attendees and speakers made a unique opportunity to dive into the topic from Findwise perspective.

Dive into Big Data

In the Opening Keynote, presented by Program Chairman PhD Geert Verstraeten, we could hear about ways to increase the impact of Predictive Analytics. Unsurprisingly a lot of fuzz is about embracing Big Data.  As analysts have more and more data to process, their need for new tools is obvious. But business will cherish Big Data platforms only if it sees value behind it. Thus in my opinion before everything else that has impact on successful Big Data Analytics we should consider improving business-oriented communication. Even the most valuable data has no value if you can’t convince decision makers that it’s worth digging it.

But beeing able to clearly present benefits is not everything. Analysts must strive to create specific indicators and variables that are empirically measurable. Choose the right battles. As Gregory Piatetsky (data mining and predictive analytics expert) said: more data beats better algorithms, but better questions beat more data.

Finally, aim for impact. If you have a call center and want to persuade customers not to resign from your services, then it’s not wise just to call everyone. But it might also not be wise to call everyone you predict to have high risk of leaving. Even if as a result you loose less clients, there might be a large group of customers that will leave only because of the call. Such customers may also be predicted. And as you split high risk of leaving clients into “persuadable” ones and “touchy” ones, you are able to fully leverage your analytics potencial.

Find it exciting

Greatest thing about Predictive Analytics World 2012 was how diverse the presentations were. Many successful business cases from a large variety of domains and a lot of inspiring speeches makes it hard not to get at least a bit excited about Predictive Analytics.

From banking and financial scenarios, through sport training and performance prediction in rugby team (if you like at least one of: baseball, Predictive Analytics or Brad Pitt, I recommend you watch Moneyball movie). Not to mention Case Study about reducing youth unemployment in England. But there are two particular presentations I would like to say a word about.

First of them was a Case Study on Predicting Investor Behavior in First Social Media Sentiment-Based Hedge Fund presented by Alexander Farfuła – Chief Data Scientist at MarketPsy Capital LLC. I find it very interesting because it shows how powerful Big Data can be. By using massive amount of social media data (e.g. Twitter), they managed to predict a lot of global market behavior in certain industries. That is the essence of Big Data – harness large amount of small information chunks that are useless alone, to get useful Big Picture.

Second one was presented by Martine George – Head of Marketing Analytics & Research at BNP Paribas Fortis in Belgium. She had a really great presentation about developing and growing teams of predictive analysts. As the topic is brisk at Findwise and probably in every company interested in analytics and Big Data, I was pleased to learn so much and talk about it later on in person.

Big (Data) Picture

Day after the conference John Elder from Elder Research led an excellent workshop. What was really nice is that we’ve concentrated on the concepts not the equations. It was like a semester in one day – a big picture that can be digested into technical knowledge over time. But most valuable general conclusion was twofold:

  • Leverage – an incremental improvement will matter! When your turnover can be counted in millions of dollars even half percent of saving mean large additional revenue.
  • Low hanging fruit – there is lot to gain what nobody else has tried yet. That includes reaching for new kinds of data (text data, social media data) and daring to make use of it in a new, cool way with tools that weren’t there couple of years ago.

Plateau of Productivity

As a conclusion, I would say that Predictive Analytics has become a mature, one of the most useful disciplines on the market. As in the famous Gartner Hype, Predictive Analytics reached has reached the Plateau of Productivity. Though often ungrateful, requiring lots of resources, money and time, it can offer your company a successful future.

Data and Search Going Big?

Posted on by
1

A few enterprise search specialists from Findwise recently attended the Scandinavian Developer Conference 2012. One of the tracks was Big Data, which is very much related to search. It had some interesting talks about how to handle large amounts of data in an efficient way. Special thanks to Theo Hultberg, Jim Webber and Tim Berglund!

The theme was that you should choose a storage system which is well suited for the task. This may seem like an obvious point, but for a long time this was simply ignored; I’m talking about the era of relational databases. Don’t get me wrong, sometimes a relational database is the very best for the job, but in many cases it isn’t.

Data is jagged by nature, i.e. not all objects have the same properties. This is why we shouldn’t force them to fit into a square table, instead everything should be denormalized! The application accessing the data will be aware of the information structure and will handle it accordingly. This will also avoid expensive assembly operations (such as joins) to get the data in the format we want when retrieving it. Why should you split up your data if you are going to assemble it over and over again? Also remember that disk space is cheap, pre-compute as much as possible. The design of a Big Data system should be governed by how the data will be retrieved.

Another step away from the relational databases is the relaxation of some of the ACID properties: Atomicity, Consistency, Isolation and Durability. Again, this is along the lines of choosing the components best suited for the system. Decide which properties are a must have and which are not so important.

Relaxing the ACID properties, such as consistency, can give great performance gains. The NoSQL database Cassandra is eventually consistent and its write performance scales linearly up to 288 nodes (and probably even higher) which gives a write performance of over 1 million writes per second!

However, relaxation of these properties is not a new concept in the world of search engines. When indexing a document, it will usually take a number of seconds before it is searchable. This is called eventual consistency, i.e. the state of the search engine will be brought from one valid state to another, within a sufficiently long period of time. Do we really need documents that were just submitted to the search engine to be
searchable instantly? Most likely, no. Isolation is another property that is not crucial to a search engine. Since a document in an index doesn’t have any explicit relations to any other documents in the same index, there isn’t a great need for isolation. If two writes for the same document are submitted at the same time, there is probably something wrong in another part of the system.

So what does all this mean for search? There is an interesting challenge in storing jagged data in large amounts and then making good use out of it. To search in vast amounts jagged data, you need a lot of querytime field mappings (to make relevant data searchable) … or do you? There is also the issue of retaining a good relevancy model, which is absolutely vital to a search engine. How do you measure the relevance of arbitrary metadata and then weigh it all together? Maybe we need to think in new ways about relevance all together?

Whomever can solve these problems in a good way with a minimum amount of manual labor, is a name we’ll be hearing from a lot in the future.