February 2023 Patchday: EWS problems after Exchange Server security update

[German]After the installation of the February 14, 2023 security updates, I got some reports from administrators that Exchange Web Services (EWS) is causing issues. For example, the calendar update isn't working. Uninstalling the security update solved the problem. Microsoft has confirmed the issues and provided a workaround. Therefore, a brief overview of the facts – perhaps administrators from the readership can say something about it.

EWS issues after update

As of February 14, 2023 Microsoft has released the security updates for Exchange 2013 to 2019 to address various security vulnerabilities. I had reported in the blog post Exchange Server Security Updates (February 14, 2023). Comments then came in from the readership about issues with EWS (Exchange Web Services). nk writes in this German comment (I've translated it):

The reports are increasing about EWS crashes and the associated problems.

On my Ex2016 I can confirm this as well.

Server side search not working, addins not loading, scheduling wizard or calendar shares taking forever to load….

practically everything that is based on EWS only works sporadically or not at all.

And German blog reader Enno added in this comment:

EWS problems I have too and we just rolled back the update. Apparently a fix in the current patch has a deserialze issue and ends up in a "deny" which is supposed to mitigate something in the code somewhere. This is how I understand the StackTrace from the event log with ID 1325

Message: Deserialization of type System.MarshalByRefObject blocked due to InDeny at location ClientExtensionCollectionFormatter. […]

After blog reader Martin contacted me today with the following mail, I pulled out the hints in this blog post separately.

Hello Günter,

in the comments to your article: Exchange Server Sicherheitsupdates (14. Februar 2023) … there are now more and more hints that there are EWS problems. I can confirm this from our environment:

For example, access to shared calendars whose mailbox is on a system that has already been patched is also no longer possible ("Calendar could not be updated").

On the other hand, access to calendars whose mailboxes are located on unpatched servers works without any problems.

This is just an addition, in case you want to write another article about this topic.

At this point my thanks to the readers. A user with the alias nak_87 also posted some comments about issues (ASP.NET (1325) and WAS (5011) errors) in the Techcommunity article about the security updates.

EventID 5011, WAS

Fatal communication error in the Windows Process Activation Service for a process for the application pool "MSExchangeServicesAppPool". The process ID is "14692". The data field contains the error number.

Outlook search stops working. "Something went wrong…" and "Connection problem". Installed KB5023038-x64-en.exe for Exchange Server 2016. Are more users complaining about EWS problems in the comments. User enno0815de writes here:

Hi, we have the same problems with EWS on WinServer2016 (Patchlevel Jan2023) and latest CU for Exchange 2016. All clietns which uses EWS are broken, f.e. Teams, MacMail and Evolution. We rolled the update back as we need EWS. Is there any workaround already available? Regards Enno

Also here a broken EWS is reported. Also at reddit.com I've found such reports. Going back to the January 2023 patch level on Exchange Server 2016 fixes these issues. That was Microsoft's recommendation in comments to the Techcommuniy article introducing Feb. 2023 security updates.

Revisions update did not fix

In my blog post Microsoft's February 2023 Patchday: Incorrect updates in WSUS, Exchange and Windows I've reported, that Microsoft has offered old January 2023 patches for Exchange Server. In the meantime, this error has been corrected and the new update is installed. In this German comment, blog reader cram writes that with the second February 2023 update for Exchange Server 2016, there are no more EWS issues.

Exchange 2016 CU23:
All is now well again after the second update. The build version is now 15.01.2507.021 and the update shows up twice in the Windows Update history.

Yes everything is running and loaded via Windows Update. ECP and calendar are not broken.

Carsten confirms in a reply that for him "ECP, OWA, Calendar, Search, all work on Exchange 2016 CU23". But later I got reports, that the issu is still present with the 2nd update.

Microsoft provides a workaround

Microsoft is aware of the EWS issue and could reproduce it in it's labs. They have now extended the "Known Issues" section of this Techcommunity article with the following text.

After installation of February 2023 SU, some Exchange 2016 and 2019 customers can see EWS application pool crash with Event ID 4999 with the following error:

E12IIS, c-RTL-AMD64, 15.01.2507.021, 
w3wp#MSExchangeServicesAppPool, 
M.Exchange.Diagnostics, 
M.E.D.ChainedSerializationBinder.EnforceBlockReason, 
M.E.Diagnostics.BlockedDeserializeTypeException, 
437c-dumptidset, 15.01.2507.021.

The issue is causing connectivity issues to clients using the EWS protocol. We have a workaround for this (but note that events 4999 might still continue to be logged but functionality should be restored). If you are experiencing this problem, our recommendation is to use the following workaround and keep February SU installed:

1. Create the following regkey in the exchange servers:

SOFTWARE\Microsoft\ExchangeServer\v15\Diagnostics\DisableBaseTypeCheckForDeserialization

The regkey is 'string value' type and needs to have a value of 1.

2. Create the below setting override:

1. 1. New-SettingOverride -Name "Adding learning location ClientExtensionCollectionFormatter" -Server <ServerName> -Component Data -Section DeserializationBinderSettings -Parameters @("LearningLocations=ClientExtensionCollectionFormatter") -Reason "Deserialization failed"
   2. Force the application of the setting by running the following:
      Get-ExchangeDiagnosticInfo -Process Microsoft.Exchange.Directory.TopologyService -Component VariantConfiguration -Argument Refresh
      Restart IIS app pools.

Got reports from German blog readers writing, that this workaround works.

Addendum 2: The workaround helps only partially. Martin contacted me by mail on 2/17/2023 and wrote the following:

Hi Günter,

we implemented the workaround yesterday evening, but it seems to help only partially:

Outlook on Windows works fine again and also shows shared calendars, but the search is still affected.

With Outlook under macOS 13.2.1 Ventura, however, I have disconnects every few minutes, then a reconnect after a few minutes –> then it also syncs and access to calendars works, but then the connection breaks again. My colleague has the same behavior, so I can safely rule out that it only affects my machine/network connection.

The messages about macOS problems I had previously read on the Internet.

Addendum 3: See also the Microsoft support article EWS web application pool stops after the February 2023 Security Update is installed for further informationen and workarounds.

Similar articles
Exchange Server Security Updates (February 14, 2023)
Microsoft's February 2023 Patchday: Incorrect updates in WSUS, Exchange and Windows