3 easy ways to integrate external data sources with SharePoint Online

Introduction

SharePoint Online provides powerful tools able to search through various types of data. At Findwise we have worked with Microsoft search applications since the begining of the FAST era. If you have questions about or need help with integration of external sources – feel free to write me a couple of lines: lukasz.wojcik@findwise.com

 

Lets get started! First you must provide some content to SharePoint.

Here are some solutions you can choose from to feed SharePoint Online with your data:

Pushing data to SharePoint list using RESTful service

Using Business Connectivity Service

Using custom connector in hybrid infrastructure

 

Pushing data to SharePoint list using RESTful service

The most simple method to put some data in SharePoint is to write it directly to the SharePoint lists.

SharePoint Online exposes a REST API which can be used to manipulate lists.

Following steps will guide you through pushing data to SharePoint lists.

1. No token, no ride

First things first. In order to use any manipulation in SharePoint you must obtain an access token.

To do so, you must follow these steps:

  1. Handle page load event
  2. In the load event handler, read either of the following request parameters:
    • AppContext
    • AppContextToken
    • AccessToken
    • SPAppToken
  3. Create a SharePointContextToken from previously retrieved token using JsonWebSecurityTokenHandler
  4. Get the access token string using OAuth2S2SClient

2.     Know your list

By the time you want to manipulate your list you should probably have known your list name but you may not know its ID.

So, if you want to retrieve lists, you should call a GET method:

/_api/Web/lists

with header:

Authorization=Bearer <access token>

with content type:

application/atom+xml;type=entry

and accept header:

application/atom+xml

3.     Add entries to the list

Once you finally retrieve your list, you are ready to actually push your data.

There are few additional steps that need to be taken in order to execute POST request needed to add the items to the list:

  1. Get context info by calling POST method:
    /_api/contextinfo
  2. Get the form digest from received context info xml
  3. Get the list item entity type full name from the list data by calling GET method:
    /_api/Web/lists(guid'<list ID>')
  4. Form query string used to add new item to the list:
    {'__metadata':{'type':'" + <list item entity type full name> + "'}, 'Title':'" + <new item name> + "'}}
  5. Get the list items data by calling POST method:
/_api/Web/lists(guid'<list ID>')/Items

with headers:

Authorization=Bearer <access token>
X-RequestDigest=<form digest>

with content type:

application/json;odata=verbose

and accept header:

application/json;odata=verbose
  1. Write the byte array created upon the query string to the request stream.

That’s all, you’ve just added an entry to your list.

A full example code can be found here:

https://github.com/OfficeDev/SharePoint-Add-in-REST-OData-BasicDataOperations

 

 

Using Business Connectivity Service

SharePoint can gather searchable data by itself in a process called crawling. Crawling is getting all data from a specified content source and indexing its metadata.

There are various possible content sources that SharePoint can crawl using its built-in mechanisms, such as:

  • SharePoint Sites
  • Web Sites
  • File Shares
  • Exchange Public Folders
  • Line of Business Data
  • Custom Repository

In first four types of content you can choose multiple start addresses that are base paths where crawling process starts looking for data to index.

SharePoint Sites include all SharePoint Server and SharePoint Foundation sites available at the addresses specified as start addresses.

Web Sites include all sites over the Internet.

File Shares include files available via FTP or SMB protocols.

Exchange Public Folders include messages, discussions and collaborative content in Exchange servers.

Line of Business Data and Custom Repository include custom made connectors that provide any type of data. These are described in another method of connecting external data below.

To use first four types of content, all you have to do is to specify addresses where the crawling process should start its operation. Alternatively you can specify crawling schedule which will automatically start indexing data at the time specified in schedule.

There are two types of crawling:

  • Full – slower, indexes all encountered data, replacing any already existing data by new version
  • Incremental – faster, compares dates of encountered data and existing data and indexes the data only if the existing data is outdated

Though these methods are very simple and easy to use, they provide very limited flexibility and if you need more personalized way of storing your data in SharePoint which will be searchable in the future you should use more advanced technique involving creating Business Data Connectivity model, which is described below.

 

 

Using custom connector in hybrid infrastructure

Business Connectivity Service is a powerful extension but to get the most out of it, you must make some effort to prepare the Business Data Connectivity model used to define the structure of data you want to be able to search through.

1.     Create Business Data Connectivity Model

There are two simple ways to create the Business Data Connectivity model:

  • Using Microsoft SharePoint Designer
  • Using Microsoft Visual Studio

The Business Data Connectivity model is in fact stored in XML file so there’s the third way of creating the model – the hard way – edit the file manually.
Although, editing the Business Data Connectivity model file is not that easy as using visual designers, in many cases it’s the only way  to add some advanced functionalities, so it is advised to get familiar with the Business Data Connectivity model file structure.

Using Microsoft SharePoint Designer and Microsoft Visual Studio methods involve connecting to SharePoint On-Premise where the model is deployed. After the deployment the model needs to be exported to a package which can be installed on destination site.

1.1. Create Business Data Connectivity Model using Microsoft SharePoint Designer

The simplest  way to get started with Business Data Connectivity Model is to:

  • Run Microsoft SharePoint Designer
  • Connect to the destination site
  • Select External Content Types from the Navigation pane
  • Select External Content Type button from the External Content Types ribbon menu

The SharePoint Designer allows to choose the external data source from:

  • .NET assembly
  • Database connection
  • WCF web-service

The advantage of this method is that the model is automatically created from data discovered from data source.
For example if you choose database connection as a source of your data, the designer allows you to pick the database entities (such as tables, views, etc.) as a source and guides you through adding operations you want to be performed during the search process.
Saved model is automatically deployed in connected site and ready to use.

The disadvantage of this method is that only simple data types are supported and you won’t be able to add operations providing functionality of downloading attachments or checking user permissions to view searched elements,
thus adding parts responsible of these functionalities to the model file may be required.

1.2. Create Business Data Connectivity Model using Microsoft Visual Studio

In order to use the Visual Studio to create the Business Data Connectivity Model you must be running the environment on a system with SharePoint installed
To create the Business Data Connectivity Model you must take a few steps:

  • Run Visual Studio with administrative privileges
  • Create new project and select the SharePoint Project from SharePoint from either Visual C# or Visual Basic templates
  • Select Farm Solution and connect to your SharePoint site
  • Add new item to your newly created SharePoint project and select the Business Data Connectivity Model

Your new BDC Model can be now designed either in built-in SharePoint BDC Designer or in built-in XML Editor but with only one of them at time.

The advantage of designing the model in visual designer is that all defined methods are automatically generated in corresponding service source code.
Once project is built, it can be deployed directly to the connected destination site with a single click.

The disadvantage however is that you must define all fields of your business data by yourself and also create corresponding business model class.
You must also provide your connection with external system such as database.

While this method is very convenient when deploying the solution on SharePoint On-Premise, you must bear in mind that SharePoint Online doesn’t allow additional .NET assemblies that often come along with the model when creating a SharePoint Project containing a Business Data Connectivity Model.

2.     Export Business Data Connectivity Model

Once the model is created it needs to be exported to a package that can be installed on a destination site.

If you created your model in SharePoint Designer you can just right click on the model and select Export BCD model.

If you created your model in Visual Studio, you can export the model by selecting Publish command from Build menu. The simplest way to save the package is to select filesystem as a destination and then point where the package file should be saved.

3.     Import Business Data Connectivity Model into destination system

Once you have an installation package, you can import it as a solution in your SharePoint site settings.

To do so, navigate to the site settings and the to the solutions, where you can click on the Upload solution button and select the installation package.

 

Since SharePoint Online doesn’t allow using your own code as a data connector, you can use Hybrid infrastructure which involves using Business Data Connectivity Model on SharePoint Online side and the .NET assembly containing all the logic on correlated SharePoint On-Premise side. The logic provides all the necessary connections to data sources, formatting the data and all other customer required processing.

 

 

 

Conclusion

As you can see, integrating external data seem to be pretty simple and straight forward, but it still needs some effort to do it properly.

In the future posts I’ll cover the methods described above with details and some examples.