Build a SharePoint Server 2016 Hybrid Lab

SharePoint Server 2016 has been out there for a while. One big feature of it is the hybrid configuration with Office 365. To understand how it works, I built a lab environment based on Azure VMs and a trial subscription of Office 365. Here is how I did it.


To build a lab environment for hybrid solutions, you need the following components in place.

  • An Office 365 subscription. A trial is fine.
  • A public domain name. The default <yourcompany> domain that you get from the O365 subscription won’t work in hybrid scenarios. You have to register a public domain if you don’t have one.

Configure Office 365

In order to configure the hybrid environment, you must register a public domain with your O365 subscription. The process is like you go to your O365 subscription and kick start a setup process. O365 will generate a TXT value. You need to create a TXT record in the DNS of your domain vendor with that value, and then ask O365 to verify it. Once the domain is verified, the domain is register with your O365 subscription successfully. More details can be found here.

You don’t need to create those DNS records for mail exchange such as MX etc. if you just want to test SharePoint hybrid scenarios. You only need to create them if you also want to test the mailbox features.

The next step is to configuration AD sync between your on-premise AD and the Azure AD created with your O365 subscription. You can configure the Azure AD Connect tool to do it. And for a lab environment, AD sync with password sync is good enough. You can also try AD sync SSO if you have an AD FS to play with.

Before kicking start the AD sync, you might have to do some cleaning on AD attributes. I changed the following:

  • Add a valid and unique email address in the proxyAddresses attribute.
  • Ensure that each user who will be assigned Office 365 service offerings has a valid and unique value for the userPrincipalName attribute in the user’s user object.

With the cleaning done, you can start to sync the AD. You should be able to see users account in the O365 admin center after syncing.

Configure SharePoint Server 2016

Deploy the SharePoint Server 2016 farm. You can try the MinRole deployment if you have multiple servers. In my lab, I just deployed a single server.

The following service applications are required for the hybrid scenarios.

  • Managed Metadata Service
  • User Profile Service with user profile sync and MySite host.
  • App Management Service
  • Subscription Settings Service
  • Search Service for hybrid search scenario

The user profile properties need to have the following mapping:

  • User Principal Name property is mapped to userPrincipalName attribute.
  • Work email property is mapped to mail attribute.

Configure Hybrid

Once you have the O365 and SharePoint Server 2016 ready, you can start to configure the hybrid. It is fairly simple with the help of Hybrid Picker of SharePoint Online. You just need to go to SharePoint admin center of O365, click configure hybrid and pickup a hybrid solution, follow the wizard. If everything is ok, you will get the hybrid configured. Browse to an on-premise site, and you should see the app picker like the screenshot below.

Next Step

Next thing to try is to configure the server to server trust and the cloud hybrid search. Stay tuned.


A better way to control the toolbar of the XlstListViewWebPart

Sometimes you may want to programmatically control how the toolbar of an XlstListViewWebPart is shown on the page. If you search it on the web, you may find that a lot people tried to accomplish it with .NET Reflection, like what was shown in this post. It reflected into SharePoint assembly and called an internal method. Using this way may be able to achieve the goal, but it is a totally bad idea because it broke the basic .NET programing rules. When an method was declared as internal, it meant not to be called publicly. Calling it with Reflection, typically to SharePoint, may raise the supportability concerns.

So are there a better way to achieve the goal? The answer is yes. For example, to hide the toolbar of an XlstListViewWebPart, you can simply use the following 3 lines of code.

view.Toolbar = null;
view.Toolbar = @”<Toolbar Type=’None’ />”;

If you want to do other things with the toolbar, just replace the value of the 2nd line with the CAML of your toolbar. The view object in the above code is an SPView object. This way is simple, and more importantly we don’t use any internal method here.

Here we reach the end of this post. But if some of you may want to know how this way works, you can find it out by yourself with a reflector tool and checking how SPView.Toolbar property was implemented.

Exporting SharePoint search result with a custom display template

In my recent project, I got a requirement to export the search result of SharePoint 2013 to a CSV file. Initially, I think it was simple. I could probably extend the search result web part, generate the CSV on the server side and send it to the client. But when I looked at it in detail, I doubted if a search extension web part was a good idea. It could be way too heavy for such a simple requirement. It didn’t fit into our project very well too, because we are trying to leverage the new app model as much as possible, and to limit the server side customization as minimum as possible.

So I decided to explore another way, to use the display template to get the search result and to generate the CSV with JavaScript. Here are then two questions to answer.

  1. Is it possible to get all search result in display template easily?
  2. How to generate a file with JavaScript on the fly and prompt the user to download it?

Fortunately, I managed to figure out the answers of the above two questions. And I will show you how I did it in this post.

First of all, I created a control display template based on the OOTB Default Result template. I wrapped all my JavaScript code in a custom js file, exportresult.js, and uploaded it to the Style Library. I then linked the js in the display template with the following code:

    $includeScript(this.url, "~sitecollection/Style%20Library/export/exportresult.js");

I hooked my JavaScript function onto ctx.OnPostRender() so that it could be called after the page is rendered. I also added a button on top of the search result for user to trigger the exporting:

<div style="display: table-cell">
   <a id="idExportSearchResults" href="#">Export Result</a>

There is almost no useful document regarding to the JavaScript object used in the display template. So I have to try and debug the JavaScript code to figure out where the search result is stored. In the ctx.ListData object, there is a ResultTables array which stores several result tables. Based on my testing, the first item of this array is the search result shown in the search result web part. The 2nd item could be the result for refinement. So I used the following js code to export the result with the required metadata.

var propArray = ["Author", "Created", "ExternalMediaURL", "LastModifiedTime", "Path"];
var exportedResult = new Array();
// Get only the required the managed properties.
ctx.ListData.ResultTables[0].ResultRows.forEach(function (row) {
   var obj = new Object;
   for (var i = 0; i < propArray.length; i++) {
     obj[propArray[i]] = row[propArray[i]] ? row[propArray[i]] : "";

Finally, I need a way to export the data to a file with JavaScript. I actually found a very helpful post here. All credit of the way I used following goes to this post. The following code shows the idea.

var showSave;
// Only works with IE10 or later.
if (window.Blob && navigator.msSaveBlob) {
  showSave = function (data, name, mimeType) {
    resultBlob = new Blob([data], { type: mimeType });
    navigator.msSaveBlob(resultBlob, name);

With all the above, I can fulfill the requirement now. The code only works for IE10 and later. It could be extended to support other browsers. If you are interested in extending the code, feel free to fork it on Github.

Deploying a Remote Event Receiver in a Provider-Hosted App

The remote event receiver (RER) and the app event receiver are new concepts of SharePoint 2013. Although there have already been many articles and posts on the web regarding to how to create a remote event receiver, how to deploy it etc., you may still be bitten by the subtle difference when you want to deploy your provider-hosted app which has a remote event receiver, just like what I experienced recently in one of my projects. The app of my project worked all right in the dev environment. I only hit the problem when I tried to publish and deploy it with all certificate and SSL on. I’ve spent the most of my time in the past two days to figure out what is wrong. So here comes some of my findings.


An remote event receiver is an endpoint of a WCF service. But interestingly, when we create such a receiver in our app, we don’t have to either declare the endpoint in web.config or do so with runtime code. SharePoint knows how to talk to it. Given my limited WCF knowledge, I guess this would help to prevent clients other than SharePoint to figure out how to communicate with the services easily.

Although SharePoint knows the binding and protocol used to talk to the RER, it was not documented, at least based on my search. In dev environment without SSL, it worked fine. But when turning on SSL, you may see an error like the following, although the url of the service can be opened in the browser.

There was no endpoint listening at that could accept the message. This is often caused by an incorrect address or SOAP action.

Why it could happen is because SharePoint tries to talk to the service with basicHttpBinding. Without SSL, it works because the basicHttpBinding with HTTP scheme is one of the default in the protocolMapping. When turning on SSL, HTTPS scheme is not there and the endpoint cannot be accessed. So to overcome it, the basicHttpBinding with HTTPS scheme need to be added to the protocalMapping with the following settings in web.config:

        <binding name="secureBinding">
          <security mode="Transport" />
      <add binding="basicHttpBinding" scheme="https" bindingConfiguration="secureBinding" />

You may see the above code in many RER samples but may not know why it is needed, as it was only marked as something like “used by SharePoint app”. Hope you know the reason now.


In most of cases, the remote web of a provide-hosted app would use some sort of authentication method to prevent the anonymous access. Likewise in my project, I turn on Windows authentication for the remote web in IIS. But if you do so on an app with RER, you may see the following error:

The HTTP request is unauthorized with client authentication scheme ‘Anonymous’. The authentication header received from the server was ‘NTLM,Negotiate’.

What happened here is, based on the previous basicHttpBinding settings, there is no credential configured to be used for client authentication since the default clientCredentialType of basicHttpBinding is None. Obviously, when the client, here it is SharePoint, calls the service, it doesn’t provide any credential. So it cannot pass the Windows authentication.

To overcome this issue, the Anonymous authentication must be enabled on the remote web in IIS. However, I don’t want to expose the whole site anonymously. So I end up enabling both the Anonymous and Windows authentication in IIS, and configure the anonymous only on the RER service, something similar to the following configuration in web.config:

  <location path="AppEventReceiver.svc">
        <allow users="?"/>
    <compilation debug="true" targetFramework="4.5" />
    <httpRuntime targetFramework="4.5" maxRequestLength="2147483647" executionTimeout="14000" />
      <deny users="?" />

With all the above changing and configuration, I am able to deploy the provider-hosted app with the RER successfully and all functions work properly as well.

SharePoint 2013: 关于配置Claims Authentication的一切

网上关于如何在SharePoint中配置Claims based authentication的文章非常多,在SharePoint 2010时代,我自己也写过几篇。不过,这些文章大部分都只集中于一个特定的主题,比如怎么配置ADFS和SharePoint之间的Trust,怎么部署自己开发的Claim Provider等等,很少有文章就如何完整地部署一个Claim AuthN的环境,而不只是配置Web Application,还有怎么配置User Profile, My Site等等,给出一个完整的说明。我的上一个项目正好是关于Claim AuthN的,这篇文章算是对项目的一个总结。

配置Identity Provider

使用Claims based authentication的第一步,当然是配置SharePoint和Identity Provider之间的信任关系了。网上这样的文档非常多。如果是ADFS,可以参考官方文档,照着来基本错不了。其他的Identity Provider,配置起来也是大同小异,没什么可说的。我自己以前曾经写过一篇如何配置自己开发的Identity Provider的文章,也可以参考。多个Web Application是可以共用一个Trusted Identity Provider设置的,只要配置好相应的realm就可以了。为Trusted Token Issuer添加新的realm,可以用下面的PowerShell代码:

$uri = New-Object System.Uri("")
$ap = Get-SPTrustedIdentityTokenIssuer
$ap.ProviderRealms.Add($uri, "urn:SharePoint:Intranet")

配置Claim Provider

由于默认的Claim Provider其实什么都不做,一旦启用了Claims AuthN,用户的权限管理就会有麻烦。一般启用了Claims AuthN之后,都需要部署定制的Claim Provider,来做用户搜索或名字解析。关于Claim Provider的开发和部署也没有太多好说的,网上的资料很多,Codeplex上也有相应的开源项目。在Claim Provider部署成功之后,可以用下面的PowerShell命令,来关联Trusted Identity Provider和Claim Provider。

$sts = Get-SPTrustedIdentityTokenIssuer
$sts.ClaimProviderName = "ClaimProviderInternalName"

配置User Profile

配置好Web Application和Claim Provider之后,基本上Claims based authentication就可以用了。但是,如果想使用User Profile,例如想显示用户的display name,支持People Search等,就还需要配置User Profile Synchronization。

配置User Profile Synchronization Connection的时候,Athentication Provider Type需要选择Trusted Claims Provider Authentication以及相应的Trusted Identity Provider。User Profile里的Claim User Identifier属性需要map到作为Identifier的Claim上,比如Trusted Claims Provider配置为使用email地址作为Identifier,那么在User Profile里,Claim User Identifier就需要map到mail这个AD属性上。参考下面两幅图:

另外,如果想在My Site URL里避免名字冲突,或者不想将用户的Windows Logon Name显示在My Site URL里的话,可以改变User name这个属性的mapping。例如,如果想在My Site URL里使用email地址的话,可以将User name map到AD的mail属性,参考下图。

完成了上面的设置之后,执行一次Full Sync,用户的User Profile就会得到更新了。


完成了上面所有的配置之后,一个完整的,基于Claims based authentication的SharePoint环境基本上就完成了。如果想要在People Picker中隐藏默认的Windows Authentication的选项,可以执行一个简单的PowerShell命令,有详细的官方文档介绍,这里就不赘述了。

Configuring SharePoint 2013 to Use Office Web Apps

To configure SharePoint 2013 to use Office Web Apps, there are two essential references.

  1. Deploy Office Web Apps Server (
  2. Configure SharePoint 2013 to use Office Web Apps (

According to them, the configuration looks like very straight forward. However, the devil is in the details. Here, I am trying to list some issues that I experienced recently with the RTM version of SharePoint 2013 and Office Web Apps so I don’t forget them soon.

1. .NET Framework 3.5

When you install Office Web Apps on Windows Server 2012, you usually run the PowerShell script in the above articles to prepare the roles and features of Windows. That script doesn’t install .NET Framework 3.5 Core. However, at some points it looks like the Office Web Apps still needs the .NET Framework 3.5 Core. Without it, you may see the event error below.

Server Error in ‘/m’ Application

Warning: ASP.NET 4.0 – Event code: 3008 – Event ID: 1310

Could not load file or assembly ‘Microsoft.Build.Utilities, Version=, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified. (C:\Program Files\Microsoft Office Web Apps\BroadcastServices_Host\web.config line 44)

To avoid it, you can add the .NET Framework 3.5 Core feature, or run the following PowerShell script:

Add-WindowsFeature NET-Framework-Features, NET-Framework-Core

The only thing to notice is that Windows Server 2012 wasn’t shipped with .NET Framework 3.5. So you must have an Internet connection on the server so the core can be added successfully with the above method.

2. Office Web Apps Host

In case Office Web Apps don’t work after you finish the configuration and verified everything including the WOPI zone, you have to check the error log of the Office Web Apps. The default location of the log is C:\ProgramData\Microsoft\OfficeWebApps\Data\Logs\ULS.

At some points, if you see the following error:

WOPI – CreateWopiHttpRequest failing due to invalid host

It means the domain name or the host name of the SharePoint site is not trusted by Office Web Apps. You have to use the following PowerShell script to add the name into the host list.

New-OfficeWebAppsHost -Domain <String>

The name of the parameter may lead to some confusing. You may think it is the name of the domain which your SharePoint server belongs to, but it is not. For example, the name of your domain may be, but the url of the SharePoint site may be http://sharepoint. In this case, you should put “sharepoint” instead of “”.

Based on my testing, when configuring Office Web Apps over HTTP, you have to configure the host list. When configuring over HTTPS, the host list can be empty.

3. AllowOAuthOverHttp

Again, the TechNet article may lead to some misunderstanding to this option. According to the article, it looks like this option should be True when Office Web Apps are configured to use HTTP, and False when they are configured to use HTTPS. However, it is not true based on my testing. This option is more like depending on the configuration of the SharePoint site. If the SharePoint site is using HTTP, this option should be True no matter if HTTP or HTTPS is used by Office Web Apps.

If you see the “HTTPResponseCode forbidden” error in the Office Web Apps log, in most of case it is because of the incorrect setting of this option.

4. Word Doc Corrupted

At some points, you may see the following error in the Office Web Apps log.

[AppServerHost.exe] Word failed to open the document (Error Code = 0x80040266)

It is very like the Word document you are trying to open with Office Web Apps has been corrupted. You can upload a good document to test and confirm it.

One scenarios which may create such a corrupted Word doc is that you choose to create a new Word doc in SharePoint site with Office Web Apps. However, for some reason Office Web Apps don’t work normally. In this case, you get a corrupted Word doc in SharePoint document library.

SharePoint 2013 goes to public

With the announcement here, Office 2013 and SharePoint 2013 goes to public today. As an early beta participant, I can tell you that there are a lot of changes and new features in the new version and everyone who is working on Office and SharePoint should give it a try.

Here is a good start point: SharePoint 2013 for IT pros

Auto Update Statistics and Auto Create Statistics – ON or OFF for SharePoint 2010 Databases?

Update: CTS complained that some customers turned ON auto-create statistics on content databases because of this post. If you think you should turn on this option on content databases after reading this post, please let me know by leaving a comment below. The key message in this post is, in most of cases you don’t need to change the default settings of these two options on SharePoint databases. In the extreme case where you have to change them, contact CTS for support first.

If you ask any SQL DBA a question like, “should I turn on Auto Update Statistics and Auto Create Statistics on databases?” You’ll get an answer like, “Yes, definitely!” or probably “It depends”. But if we are talking about SharePoint 2010 databases, the answer is “It really depends”.

In the official document, it suggests to turn off these options and explains why they should be turned off. But what it didn’t tell is how SharePoint will maintain the statistics if the options are turned off. Therefore, it could lead to some confusions like should we turn them off on all databases or just on some databases? should we manually maintain the statistics if we turn them off? etc.

So let me try to explain how SharePoint maintains the statistics first.

Health Analyzer Rule: Databases used by SharePoint have outdated index statistics

As explained in this whitepaper, SharePoint uses this Health Analyzer rule to maintain the statistics (unfortunately the whitepaper didn’t mention the name of this rule). By default, this rule is scheduled to run everyday and it is a repairable rule, meaning it has a repair action. What this rule does is very simple. It will enumerate all SharePoint databases in the farm and check if the database implements a store procedure, proc_UpdateStatistics. If so, it will call this store procedure directly. Otherwise, it will do nothing.

So now the problem become simple. To find out on which databases we should turn on the options and on which we should turn them off, we just need to check if the database implements the store procedure. If it implements the store procedure, means it has its own way to maintain the statistics and doesn’t rely on SQL server. So the options can be turned off. Otherwise, the options should be on.

Based on my checking, the following databases implement the store procedure, and the two options are turned off by default, meaning SharePoint has its own way to maintain the statistics of these databases. These two options should be left OFF on these databases in most cases. The list is quite similar to the databases mentioned at the beginning of the above whitepaper. (I hope I’ve exhausted the SharePoint dbs.)

  • Configuration databases
  • Content databases
  • User Profile Service Application Profile databases
  • User Profile Service Application Social databases
  • Web Analytics Service Application Reporting databases
  • Web Analytics Service Application Staging databases
  • Word Automation Services databases
  • WSS_Logging

The only exception is the crawl db of the SharePoint search service. It implements the store procedure but the two options are turn on as well. I am not sure if it was intended to be set like that or what.

So let us go back to our original question, should we turn on or off the two options on a SharePoint 2010 database? My answer would be it depends on which database you are talking about. If the database knows how to maintain its own statistics, the options should be turned off and SharePoint will take care of it. Otherwise, the options should be turned on.

Details about how Document ID Service is activated

You may have found that Document ID Service is a very interesting feature. Sometimes you activate it and it just works as expected. But sometimes you activate it and it doesn’t work immediately. A typical example is to activate it on a team site and a publishing portal site. On the team site, it works immediately, but on the publishing portal site, it doesn’t work immediately. So what is the reason behind?

The reason why this feature works like that is related to how it is activated. Basically, when we activate this feature, it will check if the current site collection is a big site or not. If the site collection is not big, it will just enable the DocId on the site collection and register a work item for the Document ID Assignment timer job (DocIdAssignment). But if the site collection is too big, it will just register a DocIdEnableWorkItem for the Document ID Enable/Disable timer job (DocIdEnable). So to big site, it relies on the DocIdEnable timer job to enable the DocId. But you may find even if you run this timer job immediately after the feature is activated, the DocId is not enabled. It is because when the DocIdEnableWorkItem is added to the site by the feature activation, it has been specified to be executed 30 minutes later to avoid the potential conflict. You can check the real scheduled delivery time by querying ScheduledWorkItems table in the content db. So the DocId will only be enabled when you run DocIdEnable timer job after the scheduled delivery time. For example, suppose you activated DocId feature on a publishing portal at 8:00am, the DocIdEnableWorkItem was scheduled to be delivered at 8:30am. If you ran DocIdEnable timer job before 8:30am, nothing really happened. Only when you ran it after 8:30am say 8:35am, DocId was enabled then.

The final question is how big a site collection is big? The boundaries are maximum 1 web/site, 40 lists/web and 20 doclibs/web. A publishing portal is too big because by default it has 3 webs. The OOTB document center is OK because by default it has 1 web, ~30 lists and ~15 document libraries. The numbers are hard coded and, so far as I know, there is no way to change them.