Disturbed Cache

The Scenario
Sharepoint 2013 developed publishing site using adfs. AppFabric with CU5.
The problem
In a developed Sharepoint Site we were seeing a lot of strange issues.
Sometimes some requests weren’t finished, some things might not get loaded.

The troubleshooting
It was very hard to troubleshoot since there wasn’t any patterns to the errors initially.
In an attempt mitigate the issue we raised the cookie-length for the adfssessions from default 60min to 720.
This gave us even more issues, now the frequency of the errors were more often. While navigating for 10min an issue usually occurred twice.
When monitoring the connections with fiddler while browsing we noticed a lot of requests going to the adfs-server. When really it should only happens twice a day.
I ran a search for errors in the ULS log for same period and noticed some errors relating to tokens
Common errors were:
Token Cache: reverting to local cache to add the token….
Unexpected Exception in SPDistributedCachePointerWrapper::InitializeDataCacheFactory for usage ‘DistributedLogonTokenCache’…. Request time out
And some others in dump below

Now obviously I’ve tangled with the cache before so it's really no surprise the nasty distributed cache is involved in this mess.

Ok, so now we’ve learned that timeouts occur on Distributed Caches Logon Token Cache. Who do we fix it?
Luckily the error message actually contains a part of the solution. Blablabla … MaxBuffersize must be greater or equal….
To actually find these errors I used my LogSearcher

To check servers timeout-settings


The solution
1. Increase timeouts on DistributedLogonTokenCache and items on SecurityTokenServiceconfig

2. Stop all distributed cache services gracefully
Stop-SPDistributedCacheServiceInstance -graceful
Make sure to wait 5-10 minutes between first cache restart to the last. This so the cache actually have time to reconnect to the cachecluster and make friends with other caches. Note that cache need to be restarted from Manage Services after above command.
3. Run IISreset on all hosts.

Conclusions
These errors started occurring pretty might day one in the environment. Though it was when users actually started using the site as the problem became apparent.
The SecurityTokenServiceConfig doesn’t seem to be used the same way when using Windows Authentication. This made it impossible for the developers to troubleshoot in their environment without having their own adfs-solution.
Now a part of the problem was solved only by increasing cacheitems,channelopentimeout and requesttimeout. I would say that these settings solved the exceptions and fails.
While maxbuffersize and maxbufferpoolsize solved the Failed to add/get token errors below

First part I assume was due to the many API-connections that happened on the starter pages, big page with ajax-loading a lot of the components with rest. These might have maxed out the ItemsCache or it was due to the fact that tokens didn’t get saved in the cache properly or couldn’t be acquired in time.
Second part too many claims at a time, big or improperly formatted.
But Im just guessing here.

References:https://social.msdn.microsoft.com/Forums/en-US/e13871ce-5b6f-4a3c-bc04-fbce47870856/sharepoint-web-app-randomly-gives-302-redirect-to-reauthenticate-even-with-valid-fedauth-cookie?forum=sharepointgeneralprevious
https://habaneroconsulting.com/insights/sharepoint-2013-distributed-cache-bug
https://technet.microsoft.com/en-us/library/jj219613.aspx#finetune
https://danielthenerd.wordpress.com/2015/05/08/comprehensive-distributed-cache-and-security-token-service-config-review-for-sharepoint-2013/

Overall steps to move from Active Directory to ADFS authentication in Sharepoint 2013

This is for birds-eye perspective. I found it difficult to find resources that outlined the procedure. This omits everything that happens on the ADFS-server and focuses on sharepoint-parts.

1. Backup all contentdatabases, take snapshots of servers
2. Add relying party identifiers on ADFS-server. Add endpoints with ws_federation for all web applications that are going to use ADFS.
3. Add identity provider on sharepointserver (PS1.ConfigSPIdentifier)
4. Activate wsreply on sharepointserver
5. Add Identity provider to relevant web applications
Authentication Provider > Claims Authentication Types > Trusted Identity Provider
6. Configure new User Profiler Synchronization, use Authentication Provider type = Trusted Claims > your adfs
Configure User properties >
email = mail
Claim User Identifier = mail
Run full sync
7. Convert all users on webapplications. Don’t convert searchaccounts, don’t convert authenticated users. Change it to for example domain users
Use move-spuser
8. Change SuperUser and Super Reader accounts to adfs
Run change cachereaders on each webbapp and change accounts on User Policy on webapps.
9. Change loginpage for ADFS (optional, but enables automatic signin for directory users) (also enabled crawl to run as AD-user)
Copy autologin.aspx to common files\template….
Change in autologin.aspx to use current ADFS-provider
Change CentralAdministration> authpolicy > default signin page = autologin.aspx
10. Check search engine so everything works.
11. Install LDAPCP to solve peoplepicker issues.
12. Hide AD from selectionlist
13. Done.

Issues that occurred:
Couldn’t view any others mysites – solution add / permissions for all users.

Otherwise it worked quite nicely. 6000+ plus users migrated. This solution used email as primary claim. This then required the email field in AD to be populated with unique value.

Do I need to say that I take no responsbility for when this goes sideways? I don’t. But it worked for me.

PS1.ConfigSPIdentifier

$cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2("C:\xss\ADFS-Token-Signing.cer")

New-SPTrustedRootAuthority -Name "Token Signing Cert" -Certificate $cert

$map = New-SPClaimTypeMapping -IncomingClaimType "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn" -IncomingClaimTypeDisplayName "upn" -SameAsIncoming

$map2 = New-SPClaimTypeMapping -IncomingClaimType "http://schemas.microsoft.com/ws/2008/06/identity/claims/role" -IncomingClaimTypeDisplayName "Role" -SameAsIncoming

$map3 = New-SPClaimTypeMapping -IncomingClaimType "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" -IncomingClaimTypeDisplayName "EmailAddress" -SameAsIncoming

$realm = "urn:test-intranet:sharepoint"

$ap = New-SPTrustedIdentityTokenIssuer -Name "TEST_ADFS@contoso.com" -Description "STS-IP id1.contoso.com" -realm $realm -ImportTrustCertificate $cert -ClaimsMappings $map,$map2,$map3 -SignInUrl "https://id1.contoso.com/adfs/ls" -IdentifierClaim http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress

PS2.Activatewsreply

$tit = Get-SPTrustedIdentityTokenIssuer fs.contoso.com
$tit.UseWReplyParameter = $true
$tit.Update()

PS6.HideADfromLookup
$cpm = Get-SPClaimProviderManager
$ad = get-spclaimprovider -identity "AD"
$ad.IsVisible = $false
$cpm.Update()

References:
http://blogs.msdn.com/b/scicoria/archive/2011/06/10/sharepoint-2010-fba-and-sliding-sessions.aspx - Explains sliding sessions
http://msdn.microsoft.com/en-us/library/hh446526.aspx - explains adfs token
http://msdn.microsoft.com/en-us/library/office/hh147183%28v=office.14%29.aspx
http://blog.robgarrett.com/2013/05/06/sharepoint-authentication-and-session-management/ – good information on authentication process
http://www.wictorwilen.se/sharepoint-2013-with-saml-claims-and-sharepoint-hosted-apps – wsreply
http://weblogs.asp.net/wesleybakker/adfs-sharepoint-2013-single-sign-on-skip-authentication-provider-selection-page - ADFS Sharepoint 2013 Skip authentication Provider Page – autologin.aspx