Rackspace hosted Exchange suffered a devastating interruption beginning December 2, 2022 and is still continuous as of 12:37 AM December fourth. Initially described as connection and login issues, the guidance was ultimately updated to announce that they were handling a security event.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.
Customers on Buy Twitter Verified reported that Rackspace was not responding to support emails.
This has been rather the day with #Rackspace. Every hosted exchange customer has been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer independently messaged me over social networks on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Not sure the number of business that is, but it’s substantial.
They’re serving a 554 long hold-up bounce so people emailing in aren’t knowledgeable about the bounce for a number of hours.”
The main Rackspace status page provided a running upgrade of the failure but the initial posts had no details besides there was an outage and it was being examined.
The very first official update was on December 2nd at 2:49 AM:
“We are examining a problem that is affecting our Hosted Exchange environments. More details will be published as they appear.”
Thirteen minutes later Rackspace started calling it a “connection problem.”
“We are investigating reports of connectivity issues to our Exchange environments.
Users might experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates explained the ongoing issue as “connectivity and login problems” then later that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation phase” of the outage, still trying to determine what went wrong.
And they were still calling it “connection and login problems” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later on Rackspace described the situation as a “considerable failure”and began providing their customers totally free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround up until they comprehended the problem and might bring the system back online.
The official guidance stated:
“We experienced a considerable failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any more concerns while we continue work to bring back service. As we continue to work through the root cause of the concern, we have an alternate service that will re-activate your capability to send and get e-mails.
At no charge to you, we will be providing you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until more notification.”
Rackspace Hosted Exchange Security Incident
It was not up until almost 24 hours later on at 1:57 AM on December 3rd that Rackspace officially revealed that their hosted Exchange service was struggling with a security occurrence.
The statement further exposed that the Rackspace service technicians had actually powered down and detached the Exchange environment.
“After further analysis, we have determined that this is a security event.
The recognized effect is isolated to a part of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.”
Twelve hours later on that afternoon they updated the status page with more details that their security group and outdoors specialists were still working on fixing the failure.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not launched information of the security event.
A security occasion generally involves a vulnerability and there are two extreme vulnerabilities presently in the wile that were patched in November 2022.
These are the two most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack permits a hacker to read and alter information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy has the ability to run destructive code on a server.
An advisory released in October 2022 described the impact of the vulnerabilities:
“An authenticated remote assaulter can carry out SSRF attacks to intensify advantages and perform arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mailbox server, the aggressor can possibly access to other resources via lateral movement into Exchange and Active Directory site environments.”
The Rackspace blackout updates have actually not indicated what the particular issue was, only that it was a security incident.
The most current status upgrade since December fourth specified that the service is still down and customers are motivated to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in addressing the event. The accessibility of your service and security of your information is of high importance.
We have dedicated comprehensive internal resources and engaged first-rate external know-how in our efforts to minimize unfavorable effects to clients.”
It’s possible that the above kept in mind vulnerabilities belong to the security event affecting the Rackspace Hosted Exchange service.
There has actually been no statement of whether client information has been jeopardized. This event is still continuous.
Included image by Best SMM Panel/Orn Rin