Rackspace hosted Exchange suffered a catastrophic outage starting December 2, 2022 and is still ongoing as of 12:37 AM December 4th. Initially referred to as connection and login issues, the assistance was ultimately updated to reveal that they were dealing with a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be resolved.
Clients on Buy Twitter Verified reported that Rackspace was not responding to support emails.
This has been rather the day with #Rackspace. Every hosted exchange client has been down for 14 hours approximately. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not sure the number of business that is, however it’s considerable.
They’re serving a 554 long delay bounce so individuals emailing in aren’t aware of the bounce for several hours.”
The main Rackspace status page offered a running upgrade of the outage however the initial posts had no info aside from there was a failure and it was being investigated.
The first authorities upgrade was on December 2nd at 2:49 AM:
“We are examining a problem that is affecting our Hosted Exchange environments. More details will be posted as they become available.”
Thirteen minutes later Rackspace began calling it a “connection problem.”
“We are examining reports of connection concerns to our Exchange environments.
Users might experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates explained the ongoing issue as “connectivity and login concerns” then later that afternoon at 1:54 PM Rackspace announced they were still in the “examination stage” of the failure, still attempting to determine what went wrong.
And they were still calling it “connectivity and login issues” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
Four hours later Rackspace described the situation as a “considerable failure”and began providing their customers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they comprehended the problem and could bring the system back online.
The official assistance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively closed down the environment to avoid any additional concerns while we continue work to bring back service. As we continue to work through the origin of the problem, we have an alternate option that will re-activate your capability to send out and get e-mails.
At no charge to you, we will be offering you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until more notification.”
Rackspace Hosted Exchange Security Occurrence
It was not until nearly 24 hours later at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was suffering from a security event.
The announcement further revealed that the Rackspace technicians had powered down and detached the Exchange environment.
“After further analysis, we have actually identified that this is a security event.
The known effect is separated to a part of our Hosted Exchange platform. We are taking needed actions to evaluate and secure our environments.”
Twelve hours later that afternoon they updated the status page with more information that their security group and outside professionals were still dealing with solving the blackout.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not launched information of the security occasion.
A security occasion typically includes a vulnerability and there are two serious vulnerabilities presently in the wile that were covered in November 2022.
These are the 2 most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack allows a hacker to read and alter information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assailant has the ability to run malicious code on a server.
An advisory released in October 2022 explained the effect of the vulnerabilities:
“A verified remote assaulter can perform SSRF attacks to intensify advantages and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mail box server, the enemy can possibly gain access to other resources via lateral movement into Exchange and Active Directory site environments.”
The Rackspace blackout updates have actually not indicated what the particular issue was, only that it was a security incident.
The most current status update since December fourth stated that the service is still down and consumers are encouraged to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in dealing with the incident. The schedule of your service and security of your information is of high value.
We have committed substantial internal resources and engaged first-rate external proficiency in our efforts to reduce negative effects to customers.”
It’s possible that the above kept in mind vulnerabilities relate to the security event impacting the Rackspace Hosted Exchange service.
There has been no statement of whether customer details has actually been compromised. This event is still continuous.
Included image by Best SMM Panel/Orn Rin