Beyond Office 365 – knowledge graphs, Microsoft Graph & AI!

This is the first joint post in a series where Findwise & SearchExplained, together decompose Microsoft’s realm with the focus on knowledge graphs and AI. The advent of graph technologies and more specific knowledge graphs have become the epicentre of the AI hyperbole.

microsoft_graph

The use of a symbolic representation of the world, as with ontologies (domain models) within AI is by far nothing new. The CyC project, for instance, started back in the 80’s. The most common use for average Joe would be by the use of Google Knowlege Graph that links things and concepts. In the world of Microsoft, this has become a foundational platform capacity with the Microsoft Graph.

It is key to separate the wheat from the chaff since the Microsoft Graph is by no means a Knowledge Graph. It is a highly platform-centric way to connect things, applications, users and information and data. Which is good, but still it lacks the obvious capacity to disambiguate complex things of the world, since this is not its core functionality to build a knowledge graph (i.e ontology).

From a Microsoft centric worldview, one should combine the Microsoft Graph with different applications with AI to automate, and augment the life with Microsoft at Work. The reality is that most enterprises do not use Microsoft only to envelop the enterprise information landscape. The information environment goes far beyond, into a multitude of organising systems within or outside to company walls.

Question: How does one connect the dots in this maze-like workplace? By using knowledge graphs and infuse them into the Microsoft Graph realm?

Office 365 MDM

The model, artefacts and pragmatics

People at work continuously have to balance between modalities (provision/find/act) independent of work practice, or discipline when dealing with data and information. People also have to interact with groups, and imaged entities (i.e. organisations, corporations and institutions). These interactions become the mould whereupon shared narratives emerge.

Knowledge Graphs (ontologies) are the pillar artefacts where users will find a level playing field for communication and codification of knowledge in organising systems. When linking the knowledge graphs, with a smart semantic information engine utility, we get enterprise-linked-data that connect the dots. A sustainable resilient model in the content continuum.

Microsoft at Work – the platform, as with Office 365 have some key building blocks, the content model that goes cross applications and services. The Meccano pieces like collections [libraries/sites] and resources [documents, pages, feeds, lists] should be configured with sound resource descriptions (metadata) and organising principles. One of the back-end service to deal with this is Managed Metadata Service and the cumbersome TermStore (it is not a taxonomy management system!). The pragmatic approach will be to infuse/integrate the smart semantic information engine (knowledge graphs) with these foundation blocks. One outstanding question, is why Microsoft has left these services unchanged and with few improvements for many years?

The unabridged pathway and lifecycle to content provision, as the creation of sites curating documents, will be a guided (automated and augmented [AI & Semantics]) route ( in the best of worlds). The Microsoft Graph and the set of API:s and connectors, push the envelope with people at centre. As mentioned, it is a platform-centric graph service, but it lacks connection to shared narratives (as with knowledge graphs).  Fuzzy logic, where end-user profiles and behaviour patterns connect content and people. But no, or very limited opportunity to fine-tune, or align these patterns to the models (concepts and facts).

Akin to the provision modality pragmatics above is the find (search, navigate and link) domain in Office 365. The Search road-map from Microsoft, like a yellow brick road, envision a cohesive experience across all applications. The reality, it is a silo search still 😉 The Microsoft Graph will go hand in hand to realise personalised search, but since it is still constraint in the means to deliver a targeted search experience (search-driven-application) in the modern search. It is problematic, to say the least. And the back-end processing steps, as well as the user experience do not lean upon the models to deliver i.e semantic-search to connect the dots. Only using the end-user behaviour patterns, end-user tags (/system/keyword) surface as a disjoint experience with low precision and recall.

The smart semantic information engine will usually be a mix of services or platforms that work in tandem,  an example:

  1. Semantic Tools (PoolParty, Semaphore)
  2. Search and Analytics (i3, Elastic Stack)
  3. Data Integration (Marklogic, Biztalk)
  4. AI modules (MS Cognitive stack)

In the forthcoming post on the theme Beyond Office 365 unpacking the promised land with knowledge graphs and AI, there will be some more technical assertions.
View Fredric Landqvist's LinkedIn profileFredric Landqvist research blog
View Agnes Molnar's LinkedIn profileAgnes Molnar SearchExplained

.

Reflection, part 2

Some time ago I was writing about the Reflection mechanism in .NET Framework.

This time I will show you a use case, where it’s better NOT TO USE the Reflection.

Introduction

In the previous post about the Reflection I mentioned some doubt thoughts about using this mechanism and one of them has actually its justification.

So, when it’s better not to use Reflection and why?

Consider a method that accepts some objects and we want to access some property on these objects inside this method.

private void MyUniversalMethod(object obj)
{
    if (obj.GetType().GetProperty("MyPropertyName ") is System.Reflection.PropertyInfo myProperty) //Check if our object actually has the property we're interested in.
    {
        var myPropertyValue = myProperty.GetValue(obj); //Get the property value on our object.
        myProperty.SetValue(obj, new object()); //Set the property value on our object.
    }
} 

Although, technically, there’s nothing wrong with this approach, it should be avoided in most cases, because it totally breaks the concept of strong typing.

How do we do it properly?

If we are in control over the classes we are using, we should always extract the property we want to access in such method to an interface.

interface IMyInterface
{
    object MyProperty { get; set; }
} 

So our method will look a lot simpler and, what’s most important, the compiler upholds the code integrity so we don’t have to bother if the property doesn’t exist or is inaccessible on our object, because the interface forces the accessibility for us:

private void MyUniversalMethod(IMyInterface obj)
{
    var myPropertyValue = obj.MyProperty; //Get the property value on our object.
    obj.MyProperty = new object(); //Set the property value on our object.
} 

But, what if we have no control over the classes?

There are scenarios where we have to use someone else’s code and we have to adapt our code to the already existing one. And, what’s worse, the property we are interested in is not defined in any interface but there are several classes that can contain such property.

But then, it is still recommended that we don’t use the Reflection in that case.

Instead of that, we should filter the objects that come to our method to specific types that actually contain the property we are interested in.

private void MyUniversalMethod(object obj)
{
    if (obj is TheirClass theirClass) //Check if our object is of type that has the property we're interested in. If so, assign it to a temporary variable.
    {
        var theirPropertyValue = theirClass.TheirProperty; //Get the property value on our object.
        theirClass.TheirProperty = new object(); //Set the property value on our object.
    }
} 

There’s an inconvenience in the example above that we have to specify all the types that might contain the property we are interested in and handle them separately but this protects us from cases where in different classes a property of the same name is of a different type. Here we have the full control over what’s happening with strongly typed property.

Then, what is the Reflection good for?

Although I said it is not recommended in most cases, there are, however, cases where the Reflection approach would be the preferred way.

Consider a list containing names of objects represented by some classes.

We create a method that will retrieve the name for us to display:

private string GetName(object obj)
{
    var type = obj.GetType();
    return (type.GetProperty("Name") as System.Reflection.PropertyInfo)? //Try to get the "Name" property.
        .GetValue(obj)? //Try to get the "Name" property value from the object.
        .ToString() //Get the string representation of the value, if it's string it just returns its value.
        ?? type.Name; //If the tries above fail, get the type name.
}

The property “Name” is commonly used by many classes, though it’s very rarely defined in an interface. We can be also almost certain that it will be a string. We can then just look for this property by a Reflection and in case we didn’t find it use the type name. This approach is commonly used in Windows Forms PropertyGrid’s collection editors.

Use of dynamic keyword

At the point we are certain we don’t want to rely on strong typing, we can access the properties at runtime in an even simpler way, by using the dynamic keyword, which introduces the flexibility of duck typing.

private void MyUniversalMethod(dynamic obj)
{
    var theirPropertyValue = obj.TheirProperty; //Get the property value on our object.
    obj.TheirProperty = new object(); //Set the property value on our object.
} 

This is very useful in cases where we don’t know the type of the object passed to the method at the design time. It is also required by some interop interfaces.

But be careful what you are passing to this method, because in case you try to access a member which doesn’t exist or is inaccessible you will get a RuntimeBinderException.

Note that all members you will try to access on a dynamic object are also dynamic and the IntelliSense is disabled for them – you’re on your own.

XRANK in SharePoint Search REST API

I work with SharePoint Search from some time now. Since many clients need assistance on Search optimization KQL is one of my best mates. Especially XRANK is very powerful function that leverage KQL capabilities but also enlarge its complexity. Anyway I feel quite sure about what we can achieve using KQL and how. However last week a colleague of mine asked me about what is proper syntax of XRANK in REST search query…and I was like “emmm…”.

There are many not obvious questions – which characters need to be encoded? Is the syntax the same as in common KQL query?

I did quick documentation check as well as googling for an answer but there was no satisfying results at all (if there is no answer in Stack Overflow the web contains no answer).

So this post is about clarification for XRANK syntax in REST API calls.

Use Search Query Tool

The old sentence says “Do not break open doors”. That’s why I did not investigate topic by myself trying different REST queries to SP Search. Instead I used great great great tool called Search Query Tool. It really makes your work with search easier and faster. You can build any kind of KQL query in it and it will be translated to REST query because it uses it to communicate with SharePoint.

So for instance if you want to execute following KQL query

*  XRANK(cb=1) Position:Manager

Its REST equivalent will be:

<SearchEndpointURL>?querytext=’*+XRANK(cb%3d1)+Position:Manager’

As you can see syntax is the same as in common KQL query however ‘=’ character has been encoded to URI format in order to be properly understood by browser and endpoint and any spaces has been replaced by “+”.

Complex XRANK queries

Remember that in order to build you must remember about proper use of parenthesis. For instance if you want to make multiple XRANK boosts you need to arrange them in following way:

(SearchQuery XRANK(cb=1) condition1) XRANK(cb=1) condition2

In other words, if you want to add boosting for position AND for date freshness your KQL will look like below:

(* XRANK(cb=1) Position:Manager) XRANK(cb=0.5) LastModifiedTime>{Today-365}

and your REST query text will be like following:

querytext='(*+XRANK(cb%3d1)+Position:Manager)+XRANK(cb%3d0.5)+LastModifiedTime>{Today-30}’

which gives you following results:

  • results older than 30 days and for person that position does not contain “Manager” in its name will get 0 ranking points
  • results modified less than 30 days ago and for person that position does contain “Manager” in its name will get 0.5 ranking points
  • results older than 30 days and for person that position does contain “Manager” in its name will get 1 ranking points
  • results modified less than 30 days ago and for person that position does not contain “Manager” in its name will get 1.5 ranking points

 

Hope it helps you in using XRANK and KQL in REST API queries.

 

Thanks & have a great day!

Reflection… is like violence

“If it doesn’t solve your problems, you are probably not using enough of it”

Introduction

What is Reflection?

You can think of your class like it is your body. The code contained in it doesn’t “know” anything about its contents (unless it’s statically referenced, but I wouldn’t consider it “knowing about”), just like you don’t know what is inside your body.

The Reflection is then like an encyclopedia of a human body that contains all the information about it. You can use it to discover all the information about all your body parts.

So, what is that Reflection in computer programming?

The Reflection is a mechanism that allows you to discover the information about what your class is made of.

You can enumerate in the runtime through Properties, Methods, Events and Fields, no matter if they are static (Shared in Visual Basic) or instance, public or non-public.

But it does not limit you to discover information about only your classes. You can actually get the information about all the classes currently loaded into your application domain.

This means, Reflection gives you access to the places you can’t get to in a “normal” way.

The purpose of using reflection

Some say that there’s no point in getting access to class members through Reflection because:

  • Public members are accessible in the “normal” way, so using Reflection just breaks the strong typing – the compiler will no longer tell you if you are doing something wrong;
  • Accessing non-public members is just hacking.

They may be right, in both cases, but

  • Sometimes it’s way more convenient to let the program discover members in runtime so we don’t need to explicitly access them one by one, it’s more refactoring-proof, sometimes it’s the only way to implement a well-known design pattern;
  • And sometimes, hacking is the only reasonable way of achieving a goal.

Still in doubt? Let’s take a look at some examples below that describe when to use Reflection and get a profit and when it’s more like a bad practice…

Use-case 1: Enumerate your own members

One of the most popular use cases of Reflection is when you have to serialize your class and send such prepared data over the net e.g. using JSON format. A serializer (a class containing one essential method responsible for creating a data ready to send out of your class object and probably second method to restore the data back to your class object) does nothing else than enumerate through all the properties contained by the class and reads all the data from the class object and then formats it to prepare desired format (such as JSON).

Static typing approach – specify properties explicitly when assembling the output string

Now imagine you have to serialize a class without using the reflection. Let’s take a look at the example below:

Suppose we have a model class

public class SampleClass
{
    public int Id { get; set; }
    public string Name { get; set; }
}

And we want to serialize an instance of this class to a comma separated string.

Sounds simple, so let’s do it. We create a serializer class:

public static class SampleClassSerializer
{
    static string Serialize(SampleClass instance)
    {
        return $"{obj.Id},{obj.Name}";
    }
} 

Looks simple and straightforward.

But what if we need to add a property to our class? What if we have to serialize another class? All these operations require to change the serializer class which involves additional frustrating work that needs to be done each time something changes in the model class.

Reflection approach – enumerate properties automatically when assembling the output string

How to do it properly, then?

We create a seemingly bit more complicated serializer method:

 

//Make the method generic so object of any class can be passed.
//Optionally constrain it so only object of a class specified in a constraint,
//a class derived from a class specified in a constraint
//or a class or an interface implementing an interface specified in a constraint can be passed.
public static string Serialize<T>(T obj) where T : SampleClass
{
    //Get System.Type that the passed object is of.
    //An object of System.Type contains all the information provided by the Reflection mechanism we need to serialize an object.
    System.Type type = obj.GetType();

    //Get an IEnumerable of System.Reflection.PropertyInfo containing a collection of information about every discovered property.
    IEnumerable<System.Reflection.PropertyInfo> properties = type.GetProperties()
    //Filter properties to ones that are not decorated with Browsable(false) attribute
    //in case we want to prevent them from being discovered be the serializer.
        .Where(property => property.GetCustomAttributes(false).OfType<BrowsableAttribute>().FirstOrDefault()?.Browsable ?? true);

    return string.Join(",", //Format a comma separated string.
        properties.Select(property => //Enumerate through discovered properties.
        {
            //Get the value of the passed object's property.
            //At this point we don't know the type of the property but we don't care about it,
            //because all we want to do is to get its value and format it to string.
            object value = property.GetValue(obj);

            //Return value formatted to string. If a custom formatting is required,
            //in this simple example the type of the property should override the Object.ToString() method.
            return value.ToString();
        }));
} 

The example above shows that adding a few simple instructions to the serializer makes us we can virtually forget about it. Now we can add, delete or change any of the properties, we can create new classes and still Reflection used in the serializer will still discover all the properties we need for us.

With this approach all the changes we make to the class are automatically reflected by the serializer and the way of changing its behavior for certain properties is to decorate them with appropriate attributes.

For example, we add a property to our model class we don’t want to be serialized. We can use the BrowsableAttribute to achieve that. Now our class will look like this:

public class SampleClass
{
    public int Id { get; set; }

    public string Name { get; set; }

    [Browsable(false)]
    public object Tag { get; set; }
} 

Note that after GetProperties method in serializer we are using Where method to filter out properties that have Browsable attribute specified and set to false. This will prevent newly added property Tag (with [Browsable(false)]) from being serialized.

Use-case 2: Enumerate class’ methods

This is an academic example. Suppose we are taking a test at a university where our objective is to create a simple calculator that processes two decimal numbers. We’ve been told that to pass the test, the calculator must perform addition and subtraction operations.

So we create the following Form:

There are two rows, each one uses different approach to the problem. The upper row uses common “academic” approach which involves the switch statement. The lower row uses the Reflection.

The upper row consist of following controls (from left to right): numericUpDown1, comboBox1, numericUpDown2, button1, label1

The lower row consist of following controls (from left to right): numericUpDown3, comboBox2, numericUpDown4, button2, label2

In a real case we would create only one row.

Static typing approach – use the switch statement to decide which method to use based on user input

Next, to make it properly, we create a class containing some static arithmetic methods that will do some math for us:

public class Calc
{
    public static decimal Add(decimal x, decimal y)
    {
        return x + y;
    }

    public static decimal Subtract(decimal x, decimal y)
    {
        return x - y;
    }
} 

And then we are leveraging our newly created class. Since it’s an academic test, we don’t want to waste precious time on thinking how to do it properly, so we are choosing the “switch” approach:

private void button1_Click(object sender, EventArgs e)
{
    switch (comboBox1.SelectedItem)
    {
        case "+":
            label1.Text = Calc.Add(numericUpDown1.Value, numericUpDown2.Value).ToString();
            break;

        case "-":
            label1.Text = Calc.Subtract(numericUpDown1.Value, numericUpDown2.Value).ToString();
            break;

        default:
            break;
    }
} 

Plus WinForms Designer generated piece of code that populates the operation combobox:

this.comboBox1.Items.AddRange(new object[] 
{
    "+",
    "-"
}); 

The code is pretty simple and self-explanatory.

But then, the lecturer says that if we want an A grade, the calculator must perform multiplication and division operations as well, otherwise, we will get B grade at most.

Additional hazard (by hazard I mean a potential threat that may in some circumstances lead to occurrence of unhandled exception or other unexpected behavior) in this approach is that we are using “magic constants”, the “+”,“-” strings. Note that for each arithmetic operation one instance of the string is generated by the designer and another instance is used in each case statement.

If by any reason we decide to change one instance, the other one will remain unchanged and there’s even no chance that IntelliSense will catch such misspell and for compiler everything is fine.

But, apart from that, let’s hope that the lecturer will like the “+”,“-“ strings, we need to implement two additional arithmetic operations in order to get our scholarship.

So, we implement two additional methods in our little arithmetic class:

public static decimal Multiply(decimal x, decimal y)
{
    return x * y;
}

public static decimal Divide(decimal x, decimal y)
{
    return x / y; //Since we are using decimals instead of doubles, this can produce divide by zero exception, but error handling is out of scope of this test.
} 

But obviously that’s not enough. We have to add two another cases in our switch statement:

case "×":
    label1.Text = Calc.Multiply(numericUpDown1.Value, numericUpDown2.Value).ToString();
    break;

case "÷":
    label1.Text = Calc.Divide(numericUpDown1.Value, numericUpDown2.Value).ToString();
    break; 

and two another symbols in the combobox Items collection, so the WinForms Designer generated piece of code looks like this:

this.comboBox1.Items.AddRange(new object[] 
{
    "+",
    "-",
    "×",
    "÷"
}); 

We must also bear in mind changing used method names if we copy/paste the existing piece of code, it can be a real hell if we have to add multiple methods or if there are multiple usages.

Reflection approach – enumerate class methods and let user choose one of them

So, how can we do it better?

Instead of using evil switch statement we use Reflection.

First, we discover our arithmetic methods and populate the combobox with them – literally, the combobox items are the System.Reflection.MethodInfo objects which can be directly invoked. This can be done in form constructor or in form Load event handler:

private void Form1_Load(object sender, EventArgs e)
{
    //Populate the combobox.
    comboBox2.DataSource = typeof(Calc).GetMethods(BindingFlags.Static | BindingFlags.Public).ToList();

    //Set some friendly display name for combobox items.
    comboBox2.DisplayMember = nameof(MethodInfo.Name);
} 

Then in the “=” button Click event handler we just have to retrieve the System.Reflection.MethodInfo from selected combobox item and simply call its Invoke method which will directly invoke desired method:

private void button2_Click(object sender, EventArgs e)
{
    label2.Text =                              //Set the result directly in the label's Text property.
        ((MethodInfo)comboBox2.SelectedValue)? //Direct cast the selected value to System.Reflection.MethodInfo,
                                               //we have the control over what is in the combobox, so we are not expecting any surprises.
        .Invoke(                               //Invoke the method.
            null,                              //Null for static method invoke.
            new object[] { numericUpDown3.Value, numericUpDown4.Value }) //Pass parameters in almost the same way as in normal call,
                                                                         //except that we must pack them into the object[] array.
        .ToString();
} 

That’s all. We reduced whole switch, designer populated list with symbols and magic strings to 3 simple lines.

Not only the code is smaller, its refactoring tolerance is greatly improved. Now, when lecturer says that we need to implement two another arithmetic operations, we simply add the static methods to our arithmetic class (as stated before) and we’re done.

However, this approach has a hazard. The usage in button2
Click event handler assumes that all arithmetic methods in Calc class accept only two parameters of type that is implicitly convertible from decimal.

In other words, if someone add a method that processes 3 numbers, the operation represented by this method will show up in the combobox, but attempt to execute it will result in target invocation exception.

But if we tried to use it in the previous approach, we would get the error as well. The difference is that it would be a compile time error saying that we are trying to use method that accepts 3 parameters but we have provided only 2 arguments.

That’s one advantage of static typing, the compiler upholds the code integrity and the IntelliSense will inform us about any misusages long before we even hit the compile button. When we are using dynamic invoke, we must predict any exceptions and handle them ourselves.

Conclusion

The examples above describe how the Reflection mechanism can improve code maintainability and robustness. They also show that Using Reflection, though it’s handy, lays more responsibility on the developer as checking for compatibility is moved from design time to run time and the compiler will no longer lend its hand when invoking dynamically typed members.

In this article we learned how can we use the Reflection to retrieve data from properties and how to invoke methods that are now explicitly specified in design time – they can change over time but the code that are using them will remain unchanged.

In the next article I’ll show you more advanced usages of the Reflection such as storing data, accessing private members and instantiating classes on some interesting examples like Building An Object From Database or Implementing Plugin System. You will also eventually find out why Reflection is like violence

SharePoint Framework vs Sharepoint apps vs Sharepoint solutions

  • “I’m so confused with all this SharePoint Framework, apps, solutions…I just wanted to develop for Sharepoint!”
  • “What can I use SharePoint Framework (SPFx) for? Can I use it for branding?”
  • “When should I pick custom SharePoint solution over SharePoint Framework or SharePoint App?
  • “How can I make elevated privilages in sharepoint hosted app?”

During our years of experience in SharePoint development we’ve seen those questions many times. They were asked by IT devs of our clients, by users of tech blogs/forums and also by ourselves (yes, we’re learning all the time!). Since you’re here we assume that you are confused a little bit too but don’t worry. We know what you feel. That’s why we’ve created this post.

At Findwise we work with improving the SharePoint experience on a daily basis. If you want to know more about our areas and offers visit our site.

Continue reading

The search experience in SharePoint 2013: customised or targeted?

This post is the fourth in a series of four articles providing several best practices on how to implement and customise the search experience in SharePoint 2013. The previous posts listed the differences between the cloud and on-premise SharePoint, provided considerations when upgrading to SharePoint 2013, and dealt with the practicalities of configuring search in SharePoint Online. This fourth post handles the more advanced topic of ranking results and the future of search in SharePoint.

Managing ranking

We’ve previously mentioned the query rules as a way to change the ranking of the search results based on your requirements. These allow the promotion of certain search results or search result blocks on top of the ranked searched results, and more advanced query rules allow even changing the ranking of the search results based on what the query terms are.

By using query rules, customising the search results web part, and a few content by search web parts, you can change the behaviour of the search depending on what user is accessing it. That is, you would also need good metadata to make this work, but having a complete user profile (including the job title, department, and interests) is a good start. Based on such user information, you can define how the search experience for that user will be.

Changing ranking using query rules, however, requires a query rule condition, which describes the prerequisites that the query must fulfil in order for the query rule to fire. For changing the results for all queries, you can use the next approach.

If the default ranking does not satisfy your search requirements and you want to change the order of the ranked search results, SharePoint provides the possibility of changing the ranking models. It is a feature available in SharePoint Online as well, as described in the TechNet documentation: “SharePoint Online customers need to download and install the free Rank Model Tuning App in order to create and customize ranking models.”

A ranking model contains the features and corresponding weights that are used in calculating a score for each search result. Changing the ranking models might require a deeper and theoretical knowledge of how search works, and those that take the challenge of changing the ranking model are often dedicated search administrators or external specialised consultants.

The Ranking Model Tuning app is free on the App Store - http://office.microsoft.com/en-001/store/ranking-model-tuning-WA104192565.aspx

The Ranking Model Tuning app is free on the App Store

The Rank Model Tuning App provides a user interface for creating custom ranking models, and can be used for both SharePoint Online and SharePoint Server, though in SharePoint 2013 Server there is also the possibility to use PowerShell to customise ranking models. New models are based on existing ranking models for which you can add or remove new rank features and tune the weight of a rank feature. It also allows for evaluating the new ranking model using a test set of queries. The set of test queries can be constructed from real queries made by users that can be gathered from previous search logs, for example. How to use the tuning app is explained step-by-step in the documentation on the Office site.

Changing the weight of certain file types (say for example for PowerPoint documents compared to Excel documents) might be enough for many search implementations, but depending on the content, the features that influence the ranking of the search results can become more elaborate. For example, a property defining whether documents are either official or work-in-progress might become an important factor in determining the ranking of search results. SharePoint provides the liberty to create new properties, so it makes sense that these can be used in search to improve the relevance.

It should be pointed out, however, that changing the ranking model influences all searches that are run using that ranking model. Though the main idea of changing the ranking model is to improve the ranking, it can become much too easy to make changes that can have an undesirable effect on the ranking. This is why a proper evaluation of ranking changes needs to be part of your plan for improving search relevance.

The office graph and the future of social

The social features introduced in SharePoint 2013 provide a rich social experience, which is interconnected with the search experience. Many social features are driven by search (such as the recommendations for which people or documents to follow), and social factors also affect the search (such as finding the right expertise from conversations in your network).

In the month of June 2012 Microsoft acquired the social enterprise platform Yammer. The SharePoint Server 2013 Preview has been made available for download since July 2012, and it reached Release to Manufacturing (RTM) in October the same year. The new SharePoint 2013 implements new social features (see for example the newsfeed, the new mysites and the tagging system), many of which are overlapping with those available in Yammer! This brings us to the question on everyone’s mind since the acquisition of Yammer: what is the future of social in SharePoint? Should you use SharePoint’s social features or use Yammer?

In March 2014, Microsoft announced that they will not include new features in the SharePoint Social but rather invest in the integration between Yammer and Office 365. The guidance is thus to go for Yammer.

“Go Yammer! While we’re committed to another on-premises release of SharePoint Server—and we’ll maintain its social capabilities—we don’t plan on adding new social features. Our investments in social will be focused on Yammer and Office 365” – Jared Spataro, Microsoft Office blog

Also at the SharePoint conference this March 2014, Microsoft introduced the Office Graph, and with it Oslo as the first app demo using it. During the keynote, Microsoft mentions that the Office Graph is “perhaps the biggest idea we’ve had since the beginning of SharePoint”. The office graph maps relationships between people, the documents they authored, the likes and posts they made, and the emails they received; it’s actually an extension of Yammer’s enterprise graph. The Oslo application is leveraging the graph, in a way that looks familiar from Facebook’s graph search.

The Office Graph, connecting people and information - Microsoft Office Blog http://blogs.office.com/2014/03/03/work-like-a-network-enterprise-social-and-the-future-of-work/

The Office Graph, connecting people and information – Microsoft Office Blog

The new Office Graph provides exciting opportunities, and has consequences for how the search will be used. Findwise started exploring the area of enterprise graph search before Microsoft announced the Office Graph – see our post about the Enterprise Graph Search from January 2013.

Reluctant to go for the cloud?

Microsoft has hinted during the SharePoint conference keynote in March that they will be adding new functionalities to the cloud version first. Although they are still committed to another version of SharePoint server, new updates might come at a slower pace for the on-premise version. However, Microsoft also announced that with the SharePoint SP1 there is a new functionality in the administrative interface: a hybrid setting which allows you to specify whether you want the social component in the cloud/Yammer, or your documents on OneDrive, so that you don’t need to move everything to the cloud overnight.

Let us know how far you’ve come with your SharePoint implementation! Contact us if you need help in deciding which version of SharePoint to choose, need help with tuning search relevance, have questions about improving search, or would like to work with us to reach the next level of findability.

Customizing search in SharePoint Online

Search in SharePoint 2013 – Part 3: Customizing search in SharePoint Online

This post is the third in a series of four articles providing several best practices on how to implement and customise search in SharePoint. In the first post, we provided a brief overview of the differences in terms of search between the on-premise and cloud versions, and in the second blog post we discussed several things you should consider when migrating to the new SharePoint. In this post, we will mention several search features that can be configured in SharePoint Online, and we will be specifically referring to those available in the Enterprise Plan. If you need more information than is provided in this blogpost, feel free to visit our website or contact us!

Here is a summary of what customisations for search in SharePoint Online will be discussed:

  • Defining your own custom result sources, and hiding any that you are not using
  • Setting up hybrid search if you chose a hybrid solution
  • Defining which refiners to show and how to display them
  • Adding query suggestions that are related to your organisation
  • Adding query spelling corrections
  • Changing how the search results are displayed to show previews and additional metadata

Get ready to search ‘everything’

This is the uncustomized search box that you will see on your search center page. Please note that in some SharePoint Online plans the ‘Videos’ vertical is not available.

This is the uncustomized search box that you will see on your search center page.
Please note that in some SharePoint Online plans the ‘Videos’ vertical is not available.

Everything is the default scope when performing a search in the SharePoint search center and is returning every type of result from all of your site collections. There are a few other scopes (search verticals, or so-called Result Sources) that are included by default, People, Conversations, and Videos, and these are preconfigured to search on what you would expect.

  • You can add new result sources, say for example Reports, that shows only search results that are tagged with the keyword ‘Final Report’. You define yourself what the criteria for a result source should be.
  • If there is a result source that you are not using, say for example if you have no video content and don’t plan to have in the near future, it’s less confusing for the users if you simply not show it for now. It’s easy to add it back if you will need it in the not so foreseeable future.

If you choose a hybrid solution, your content is split between the online SharePoint and the on-premise SharePoint Server.

  • It’s possible to have one search that displays results from both locations. For example, to show results from the on-premise installation in SharePoint Online, you have to define a new result source that is able to retrieve the results from the on-premise. Then you can configure the search results page to show results from both result sources (everything from SharePoint Online plus everything from SharePoint on-premise that matches the search query).

Screenshot from the post Hybrid search by the Microsoft SharePoint Team Blog showing how results from the cloud are integrated in the search results page when the user searches from an on-premises SharePoint 2013 site. Notice also the new visual refiner for date interval in the refinement panel on the left.

Screenshot from the post Hybrid Search by the Microsoft SharePoint Team Blog showing how results from the cloud are integrated in the search results page when the user searches from an on-premises SharePoint 2013 site.
Notice also the new visual refiner for date interval in the refinement panel on the left.

Drill down into the search results

The search Refiners allow the users to drill down into the search results. There is a new type of refiner in SharePoint 2013, a visual refiner, by default used for the ‘Modified Date’.

  • The way in which the visualisation of the refiners is made has drastically changed, and you can define your own visualisation of the data if you want to. For example, what about a map as a refiner, instead of a list of city names?

By default, the refiners you will see would be the Result type (example values: Excel, Web page), Author (example values: John Doe, Jane Doe), and Modified Date (shown as a distribution of values).

  • If you edit the web part responsible for the refiners, you will be able to add other refiners as well. For example, company names are automatically extracted from your content, so it is easy to simply add that to your refiners.
  • Also, another useful refiner to show to your users is the Content Type, offering one level of detail more from the Result Type refiner.

Search guidance

Query suggestions are displayed as the user types.

Query suggestions are displayed as the user types.

As the user types a query in the search box, SharePoint is able to show Query Suggestions that help complete the query. SharePoint automatically creates a list of suggestions based on previous searches. When at least 6 search results are clicked for a specific query, that query will be added to the list of suggestions.

  • Besides the list that SharePoint creates automatically, you are able to add your own list of suggestions. This is especially useful when starting fresh with your installation, since a fresh installation will come with no query suggestions. You could help the users by adding your company name, product names or similar to the initial list of suggestions. You will also find manual adding of suggestions useful when reviewing the search logs, since these can give you a new perspective on what the users are looking for, and based on that input help guide your user to the relevant results using query suggestions.
  • You are also able to import a list of suggestions that are not intended to be shown in suggestions. Say for example that your testing team uses a specific keyword for testing content. In this case, it is very probable that the test keyword will soon appear as a suggestion for all users. To avoid this, simply add the keyword to the query suggestion exclusion list.

Similar to the query suggestions, another functionality whose purpose is to help the user in formulating the query is the Query Spelling Correction. An inclusion and exclusion list is used in this case as well, the only difference is that these are managed in the Term Store, while managing query suggestions is made by importing a plain text file.

  • You can add your own terms in the query spelling correction inclusion and exclusion lists. Probably one of the most often misspelled words is the word ‘business’. Or was it ‘bussiness’? After adding this term to the list of words to be included in the spelling suggestions, the correct form of the word would be shown under the ‘Did you mean’ functionality if the user misspells it.

Change how the search results are displayed

Screenshot from an Office Blogs post showing the hover panel for a PowerPoint document.

Screenshot from an Office Blogs post showing the hover panel for a PowerPoint document.

A final item on our list of proposed customisations for your search results is to change how the search results are displayed. In SharePoint 2013, it is the Display Templates that define how each element in the search results page is displayed. For example, there is a template for the refiner, another one for the hover panel of a PDF item, another one for the hover panel of a Word item, and so on.

  • A simple fix would be make sure that you have previews for PDF files in the hover panel. It is the Office Web Apps that power the previews for Office documents (such as Word, PowerPoint, Excel), but the preview for PDF files might not be visible for you. If so, what you can do is change the display template that is associated to the PDF result type.
  • You can also define what metadata to show for each result type. For example, for a Word document you would by default be able to see the Title, a text snippet and a URL, and in the hover panel the document preview, Last Modified date and author, as well as probably a list of the main headings from the document. However, if you have added additional metadata to your document, such as Location or Keywords, you can display these in the search results as well by modifying the right display template.

You can find more information about how to administer many of these search functionalities from this Microsoft Office page and from our search experts. Let us know how far you are in implementing SharePoint online for your organisation – we sure have a few more tips to how to configure and customize the search in SharePoint!

Cloud vs. on-premise SharePoint 2013 search

Search in SharePoint 2013 – Part 1: The difference between search within on-premise SharePoint 2013 and SharePoint Online

Cloud or on-premise? Findwise offers implementation and consulting services for both scenarios. This post is the first in a series of four articles providing several best practices on how to implement and customise search in SharePoint. The focus of this first post is introducing the difference between the cloud and on-premise SharePoint 2013 in terms of search features. If you need more information than you find in this blogpost, just stop by our website or contact us

“The cloud is on fire”

That is a quote from the Microsoft Office General Manager Jared Spataro during his keynote at the SharePoint conference in Las Vegas last month. At this conference, Microsoft revealed that 60% of the Fortune top 500 adopted Office 365 in the previous 12 months. While new versions of on-premise SharePoint and Exchange Server are promised to still come next year, Microsoft is adding more and more capabilities to the cloud version.

SPC14 Keynote summary

Fun random facts about SharePoint Online presented during the keynote at the SharePoint conference in Las Vegas this year (March 3rd 2014)

In addition to the numbers above, a market analysis report done by The Radicati Group on the adoption of Microsoft SharePoint reveals that almost a quarter of the worldwide users accessing deployments of SharePoint made during the year 2013 are using the cloud based SharePoint.

When deciding whether to go for the on-premise or cloud solution, a go-to resource for your IT team is the TechNet article describing the availability of features across the solutions. That article not only divides the features between on-premise and cloud, but also between the different Office 365 and SharePoint Online plans. What is the difference? SharePoint Online is the cloud version of the SharePoint Server, but it can be deployed as a standalone service or as part of the Office 365 suite, so different plans are usually listed for these different scenarios. There are also the Office 365 Dedicated plans, but these are out of the scope for this article. The Microsoft Office site has a more business oriented comparison of the different plans, including pricing. If not decided for one or the other, there is also the possibility of a hybrid solution!

 Availability Search feature Office 365 Small BusinessOffice 365 Small Business Premium Office 365 Midsize BusinessOffice 365 Enterprise E1 or K1Office 365 Education A2Office 365 Government G1 or K1 Office 365 Enterprise E3 or E4Office 365 Education A3 or A4Office 365 Government G3 or G4 SharePoint Online Plan 1 SharePoint Online Plan 2 SharePoint Foundation 2013 SharePoint Server 2013 Standard CAL SharePoint Server 2013 Enterprise CAL
Available within all plans
Phonetic name matching Yes Yes Yes Yes Yes Yes Yes Yes
Expertise Search Yes Yes Yes Yes Yes Yes Yes Yes
Quick preview Yes Yes Yes Yes Yes Yes Yes Yes
RESTful Query API/Query OM Yes Yes Yes Yes Yes Yes Yes Yes
Result sources Yes Yes Yes Yes Yes Yes Yes Yes
Search results sorting Yes Yes Yes Yes Yes Yes Yes Yes
Ranking models Yes Yes Yes Yes Yes Yes Yes Yes
Query spelling correction Yes Yes Yes Yes Yes Yes Yes Yes
Refiners Yes Yes Yes Yes Yes Yes Yes Yes
Manage search schema Yes Yes Yes Yes Yes Yes Yes Yes
Available in all Office365 and SharePoint Online plans
Deep links Yes Yes Yes Yes Yes No Yes Yes
Event-based relevancy Yes Yes Yes Yes Yes No Yes Yes
Graphical refiners Yes Yes Yes Yes Yes No Yes Yes
Recommendations Yes Yes Yes Yes Yes No Yes Yes
Search vertical: “Conversations” Yes Yes Yes Yes Yes No Yes Yes
Search vertical: “People” Yes Yes Yes Yes Yes No Yes Yes
Query suggestions Yes Yes Yes Yes Yes No Yes Yes
Query throttling Yes Yes Yes Yes Yes No Yes Yes
“This List” searches Yes Yes Yes Yes Yes No Yes Yes
Query rules—Add promoted results Yes Yes Yes Yes Yes No Yes Yes
Avail. in Office365 Advanced Content Processing Yes Yes Yes No No Yes Yes Yes
Hybrid search No Yes Yes Yes Yes Yes Yes Yes
Query rules—advanced actions No No Yes No No No No Yes
Search vertical: “Video” No No Yes No Yes No No Yes
Not available in any of the Office 365, SharePoint Online plans
Search connector framework No No No No No No Yes Yes
Custom entity extraction No No No No No No No Yes
Extensible content processing No No No No No No No Yes

— Simplified view of the TechNet article, focusing on the search features availability across SharePoint solutions

Limitations in Office 365 and SharePoint Online plans

Is the cloud version good enough for your organisation when it comes to search features? The table above illustrates some of the things that you might be missing in terms of search, and in what follows we will discuss those whose availability varies amongst the Office 365 or SharePoint Online plans.

Query rules – advanced actions

In order to adapt the relevance of the search results to the user intent, SharePoint 2013 adds a new feature called query rules. A query rule is defined by a condition and a corresponding action to be taken when the condition is met. Within some SharePoint Online licenses, this functionality is limited to the possibility of adding promoted results, while more advanced actions are left out. The promoted results are similar to what was in previous SharePoint versions known as search keywords, or best bets, letting you promote specific results on top of the ranked search results. The more advanced actions could consist of for example changing the query or changing the ranking of the search results by promoting a certain group of results. You can read more about various usages of query rules in one of our previous blog post.

Search Connector Framework and Hybrid Search

Administrators of SharePoint Online will miss the feature of managing the different search connectors to content sources, since the search connector framework is not available. Only SharePoint content that is stored online is going to be indexed. Search results can only be retrieved from that content, or can be set up to retrieve from an Exchange Server, from a remote SharePoint, or from a search engine that uses the OpenSearch protocol. As an alternative approach to making content from other sources searchable, you can set up hybrid search. This feature is available in almost all Office 365 and SharePoint Online scenarios. It allows users to show search results from content available in the cloud and on-premise. So if you would like to index a content source that is not supported in SharePoint Online, you should be able to index it on the on-premise.

Custom Entity Extraction

The TechNet article describing features across solutions actually shows that this feature is only available with the enterprise licensing of SharePoint Server. This feature allows the extraction of custom-defined terms from your content and making them usable as search refiners. Say for example that you would like to extract all of your current product names from the content of your documents and then be able to refine your search results on the product name.

Content Processing Extensibility

The other search feature that is only available with the enterprise licensing of SharePoint Server is the content processing extensibility. In practice, this means there is an API that can be used to transform the data before it is stored in the index. For example, more advanced entity extraction can be made at this step. While the custom entity extraction discussed previously is able to identify names in the content based on a pre-defined list of names, through this API you can use a trained model to do entity extraction for example. Additional use cases could be cleaning or normalising the data according to predefined rules (for example, lowercasing all values in a property), or automatically tagging items based on the content.

It should be noted that the TechNet article is not a comprehensive list, and rather gives an overview of the major differences between solutions. Here is for example one more feature whose availability is limited.

Synonyms

One of the missing features in SharePoint Online that is available in the on-premise solution is the possibility of defining synonyms. Since it’s too easy to communicate the same thing with different words, defining synonyms or abbreviations for search phrases can help aggregate the results for the multiple ways of expressing the same information need. We hope that Microsoft will integrate this feature in the future versions of SharePoint Online as well.

Find the right documentation

When searching for which functionality is available across solutions on the Microsoft Office.com website or TechNet, make sure to check that the discussed functionality applies to your version of SharePoint. Articles usually indicate for which versions the functionality applies to.

Feature availability in MS articles

Articles on Office.com (left) and TechNet (right) indicate for which version
of SharePoint the discussed topic applies to.

Please note that things might change, new updates in SharePoint online can add functionality that was missing before.

To stay up-to-date, check the TechNet page once in a while, visit our website or contact us to help you map your requirements to the available search features across solutions.

Continuous crawl in SharePoint 2013

Continuous crawl is one of the new features that comes with SharePoint 2013. As an alternative to incremental crawl, it promises to improve the freshness of the search results. That is, the time between when an item is updated in SharePoint by a user and when it becomes available in search.

Understanding how this new functionality works is especially important for SharePoint implementations where content changes often and/or where it’s a requirement that the content should instantly be searchable. Nonetheless, since many of the new SharePoint 2013 functionalities depend on search (see the social features, the popular items, or the content by search web parts), understanding continuous crawl and planning accordingly can help level the user expectation with the technical capabilities of the search engine.

Both the incremental crawl and the continuous crawl look for items that were added, changed or deleted since the last successful crawl, and update the index accordingly. However, the continuous crawl overcomes the limitation of the incremental crawl, since multiple continuous crawls can run at the same time. Previously, an incremental crawl would start only after the previous incremental crawl had finished.

Limitation to content sources

Content not stored in SharePoint will not benefit from this new feature. Continuous crawls apply only to SharePoint sites, which means that if you are planning to index other content sources (such as File Shares or Exchange folders) your options are restricted to incremental and full crawl only.

Example scenario

The image below shows two situations. In the image on the left (Scenario 1), we are showing a scenario where incremental crawls are scheduled to start at each 15 minutes. In the image on the right (Scenario 2), we are showing a similar scenario where continuous crawls are scheduled at each 15 minutes. After around 7 minutes from starting the crawl, a user is updating a document. Let’s also assume that in this case passing through all the items to check for updates would take 44 minutes.

Continuous crawl SharePoint 2013

Incremental vs continuous crawl in SharePoint 2013

In Scenario 1, although incremental crawls are scheduled at each 15 minutes, a new incremental crawl cannot be started while there is a running incremental crawl. The next incremental crawl will only start after the current one is finished. This means 44 minutes for the first incremental crawl to finish in this scenario, after which the next incremental crawl kicks in and finds the updated document and send it to the search index. This scenario shows that it could take around 45 minutes from the time the document was updated until it is available in search.

In Scenario 2, a new continuous crawl will start at each 15 minutes, as multiple continuous crawls can run in parallel. The second continuous crawl will see the updated document and send it to the search index. By using the continuous crawl in this case, we have reduced the time it takes for a document to be available in search from around 45 minutes to 15 minutes.

Not enabled by default

Continuous crawls are not enabled by default and enabling them is done from the same place as for the incremental crawl, from the Central Administration, from Search Service Application, per content source. The interval in minutes at which a continuous crawl will start is set to a default of 15 minutes, but it can be changed through PowerShell to a minimum of 1 minute if required. Lowering the interval will however increase the load on the server. Another number to take into consideration is the maximum number of simultaneous requests, and this is a configuration that is done again from the Central Administration.

Continuous crawl in Office 365

Unlike in SharePoint 2013 Server, continuous crawls are enabled in SharePoint Online by default but are managed by Microsoft. For those used to the Central Administration from the on-premise SharePoint server, it might sound surprising that this is not available in SharePoint Online. Instead, there is a limited set of administrative features. Most of the search features can be managed from this administrative interface, though the ability to manage the crawling on content sources is missing.

The continuous crawl for Office 365 is limited in the lack of control and configuration. The crawl frequency cannot be modified, but Microsoft targets between 15 minutes and one hour between a change and its availability in the search results, though in some cases it can take hours.

Closer to real-time indexing

The continuous crawl in SharePoint 2013 overcomes previous limitations of the incremental crawl by closing the gap between the time when a document is updated and when this is visible in the search index.

A different concept in this area is the event driven indexing, which we will explain in our next blog post. Stay tuned!