Disturbed Cache

The Scenario
Sharepoint 2013 developed publishing site using adfs. AppFabric with CU5.
The problem
In a developed Sharepoint Site we were seeing a lot of strange issues.
Sometimes some requests weren’t finished, some things might not get loaded.

The troubleshooting
It was very hard to troubleshoot since there wasn’t any patterns to the errors initially.
In an attempt mitigate the issue we raised the cookie-length for the adfssessions from default 60min to 720.
This gave us even more issues, now the frequency of the errors were more often. While navigating for 10min an issue usually occurred twice.
When monitoring the connections with fiddler while browsing we noticed a lot of requests going to the adfs-server. When really it should only happens twice a day.
I ran a search for errors in the ULS log for same period and noticed some errors relating to tokens
Common errors were:
Token Cache: reverting to local cache to add the token….
Unexpected Exception in SPDistributedCachePointerWrapper::InitializeDataCacheFactory for usage ‘DistributedLogonTokenCache’…. Request time out
And some others in dump below

Now obviously I’ve tangled with the cache before so it's really no surprise the nasty distributed cache is involved in this mess.

Ok, so now we’ve learned that timeouts occur on Distributed Caches Logon Token Cache. Who do we fix it?
Luckily the error message actually contains a part of the solution. Blablabla … MaxBuffersize must be greater or equal….
To actually find these errors I used my LogSearcher

To check servers timeout-settings


The solution
1. Increase timeouts on DistributedLogonTokenCache and items on SecurityTokenServiceconfig

2. Stop all distributed cache services gracefully
Stop-SPDistributedCacheServiceInstance -graceful
Make sure to wait 5-10 minutes between first cache restart to the last. This so the cache actually have time to reconnect to the cachecluster and make friends with other caches. Note that cache need to be restarted from Manage Services after above command.
3. Run IISreset on all hosts.

Conclusions
These errors started occurring pretty might day one in the environment. Though it was when users actually started using the site as the problem became apparent.
The SecurityTokenServiceConfig doesn’t seem to be used the same way when using Windows Authentication. This made it impossible for the developers to troubleshoot in their environment without having their own adfs-solution.
Now a part of the problem was solved only by increasing cacheitems,channelopentimeout and requesttimeout. I would say that these settings solved the exceptions and fails.
While maxbuffersize and maxbufferpoolsize solved the Failed to add/get token errors below

First part I assume was due to the many API-connections that happened on the starter pages, big page with ajax-loading a lot of the components with rest. These might have maxed out the ItemsCache or it was due to the fact that tokens didn’t get saved in the cache properly or couldn’t be acquired in time.
Second part too many claims at a time, big or improperly formatted.
But Im just guessing here.

References:https://social.msdn.microsoft.com/Forums/en-US/e13871ce-5b6f-4a3c-bc04-fbce47870856/sharepoint-web-app-randomly-gives-302-redirect-to-reauthenticate-even-with-valid-fedauth-cookie?forum=sharepointgeneralprevious
https://habaneroconsulting.com/insights/sharepoint-2013-distributed-cache-bug
https://technet.microsoft.com/en-us/library/jj219613.aspx#finetune
https://danielthenerd.wordpress.com/2015/05/08/comprehensive-distributed-cache-and-security-token-service-config-review-for-sharepoint-2013/

Cachehost is null

We're getting error Cachehost is null when trying to start Distributed Cache or removing the service with remove-spdistributedcacheserviceinstance

First of all verify these settings

- Windows Firewall have AppFabric Caching Service (TCP-Out) enabled on servers running DistributedCache

- Verify DistributedCache ports open between servers running Distributed Cache : 22233-22236 and Ping/Echo

- Verify following ports for AppFabric to talk sufficiently: 445, 139,135

- Can’t start Distributed Cache from the Central Admin

- Verify server and Active Directory talking ok.

- Verify DNS working

Verify status of Cachecluster. Is AppFabric and Sharepoint Server talking ok?

- Script : compareDCvsAppFabric-------

Add-PSSnapin microsoft.sharepoint.powershell

$instancename = "SPDistributedCacheService name=AppFabricCachingService"

$DC = get-spserviceinstance |where-object {($_.service.tostring()) -eq $instancename }

write-host "Sharepoint says..."

$DC |select typename, status,id,server |format-table

Use-CacheCluster

write-host "AppFabric says...."

Get-CacheHost |format-table

Result: Something is not ok. Sharepoint says fours servers and AppFabric says two. If same servers are reporting on both DC and AF then the problem is elsewehere.

Solution:

Remove the serviceinstance from the missing servers and add them again.

Script below removes serviceinstance from the computer the script runs from!

---------script: forceremovedistcache.ps1 --------------

Add-PSSnapin microsoft.sharepoint.powershell

$instanceName ="SPDistributedCacheService Name=AppFabricCachingService"

$serviceInstance = Get-SPServiceInstance | ? {($_.service.tostring()) -eq $instanceName –and ($_.server.name) -eq $env:computername}

$serviceInstance

write-host "Deleting instance $serviceinstance.id"

$serviceInstance.Delete()

write-host "Adding distributed cache..."

Add-SPDistributedCacheServiceInstance

write-host "Done"

Scenario 2: The other way around. AppFabric says other stuff than sharepoint.

Solution:  Remove the host from the cluster.

----script: remove host from cluster ----

Add-PSSnapin microsoft.sharepoint.powershell

$cachehost =  $env:computername

Use-CacheCluster

Get-CacheHost

Unregister-CacheHost -hostname $cachehost -ProviderType SPDistributedCacheClusterProvider -connectionstring "<INSERTREGVALUE1>"

Add-SPDistributedCacheServiceInstance

Where regvalue1 is found : HKLM\SOFTWARE\Microsoft\AppFabric\V1.0\Configuration\ConnectionString  (remove “;enlist=false” from the string)

And then try to add distributed cache again.

Lessons learned:

- In a locked down environment make sure all relevant Distributed Cache ports are open before adding the service to server.

- AppFabric and Distributed Cache is not the same thing! Appfabric is a windows service while Distributed Cache is a sharepoint service instance. They don’t always agree! Make sure they’re talking!

References:

http://blogs.msdn.com/b/besidethepoint/archive/2013/03/27/appfabric-caching-and-sharepoint-2.aspx

http://www.ericjochens.com/2014/02/sharepoint-2013-distributed-cache-issues.html

http://technet.microsoft.com/en-us/library/jj891124%28v=office.15%29.aspx

http://blogs.msdn.com/b/uksharepoint/archive/2009/01/05/sharepoint-ports-proxies-and-protocols-an-overview-of-farm

http://sharepointjournal.com/2014/08/19/sharepoint-2013-distributed-cache-boon-or-bane/

http://granitestatehacker.kataire.com/2014/03/sharepoint-2013-distributed-cache-error.html

http://almondlabs.com/blog/manage-the-distributed-cache/

http://blogs.msdn.com/b/sambetts/archive/2014/03/19/sharepoint-2013-distributed-cache-appfabric-troubleshoo

http://techblog.rakasatria.com/2013/02/04/sharepoint-2013-spdistributedcacheserviceinstance-cac