Microsoft fixes wacky Exchange Y2K22 bug that disrupted email around the world


Microsoft fixes wacky Exchange Y2K22 bug that disrupted email around the world

fake images

Microsoft has released a fix for a wild Exchange Server bug that shut down local mail delivery around the world just as the clocks were ticking in the new year.

The massive outage was due to a date check failure in Exchange Server 2016 and 2019 that made it impossible for servers to adapt to the year 2022, leading some to call it the Y2K22 bug. Mail programs stored dates and times as signed integers, which peaks at 2147483647, or 231 – 1. Microsoft uses the first two numbers of an updated version to indicate the year it was released. As long as it was the year 2021 or earlier, everything worked fine.

“What the hell is Microsoft?”

However, when Microsoft released version 2201010001 on New Years Eve, the local servers crashed because they couldn’t interpret the date. As a result, messages get stuck in transport queues. Managers around the world were left desperately trying to fix problems instead of calling in the New Year with friends and family. All they had to continue were two cryptic log messages that looked like this:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

“What the heck, Microsoft !?” an administrator wrote in this Reddit thread, which was one of the first forums to report the massive failure. “On New Years Eve !? The first place I check is Reddit and you guys save my life before we even have an engineer on the phone.”

The next day, Microsoft launched a solution. It comes in two forms: an automated PowerShell script, or a manual fix in case the script didn’t work properly, some administrators reported. In either case, fixes must be made on all on-premises Exchange 2016 and Exchange 2019 servers within an affected organization. The automated script can be run on multiple servers in parallel. The software maker said the automated script “could take some time to run” and urged administrators to be patient.

The date and time verification was performed when Exchange verified the version of FIP-FS, a scanning engine that is part of Exchange’s antimalware protections. Once the versions of FIP-FS started with numbers 22, verification could not be completed and mail delivery stopped abruptly. The solution stops Microsoft’s filtering and Microsoft Exchange transport management services, removes the current AV engine files, and installs and starts a patched AV engine.

By Monday, things were back to normal for many affected organizations. It’s unclear how long the faulty date storage has been in place, but judging from the two affected versions, it was possibly introduced when Exchange Server 2016 was in development.


arstechnica.com

Leave a Reply

Your email address will not be published. Required fields are marked *