[German]Microsoft's Exchange Online experienced a malfunction (EX694090) on November 28, 2023, during which the service was no longer functional. Users were unable to receive emails via Exchange Online. However, Microsoft is trying to resolve the problem by redirecting mail deliveries. The service degradation continued until November 29, 2023, but Microsoft claims to have fixed the problem now.
Incident EX694090 on November 28, 2023
German Blog reader Andreas P. sent me the status excerpt for "Share: Incident: EX694090, impacted service: Exchange Online, impacted feature: delivery, current status is: Service degrade" (thanks for that).
Published Time: 28.11.2023 17:22:52
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: Due to an unexpected conflict, we were unable to move email traffic away from this unhealthy set of infrastructure. We reviewed the network logs to identify an alternate solution to remediate impact and discovered an authentication problem that may be contributing to this issue. We're investigating the configuration of the affected infrastructure to better understand this authentication issue, potentially allowing us to create a strategy that remediates impact.
Scope of impact: Some users expecting to receive email in any Exchange Online connection method are impacted.
Root cause: A subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact.
Next update by: Wednesday, November 29, 2023, at 1:00 AM UTC
Published Time: 28.11.2023 15:31:07
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: We've identified that a subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact. We're moving email traffic away from this unhealthy set of infrastructure to remediate impact.
Scope of impact: Some users expecting to receive email in any Exchange Online connection method are impacted.
Root cause: A subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact.
Next update by: Tuesday, November 28, 2023, at 10:30 PM UTC
Published Time: 28.11.2023 14:59:47
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.
Incident EX694090 on November 29, 2023
By November 29, 2023, Microsoft claims to have fixed the fault EX694090 through internal changes. Andreas P. has sent me the following status message excerpt.
Published Time: 29.11.2023 00:24:24
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may have not been receiving email messages in all Exchange Online connection methods.
Final status: After our review the configuration of the affected infrastructure, we've developed an expedited fix target that replaces the affected piece of infrastructure that routes the affected email messages and have confirmed through telemetry that impact has been remediated.
Scope of impact: Some users expecting to receive email messages in any Exchange Online connection method were impacted.
Start time: Tuesday, November 28, 2023, at 5:20 PM UTC
End time: Wednesday, November 29, 2023, at 4:40 AM UTC
Root cause: A subset of infrastructure that routes the affected email wasn't processing data as expected, resulting in impact.
Next steps:
– We're continuing to investigate why the subset of infrastructure that routes the affected email wasn't processing data as expected to prevent similar instances in the future.This is the final update for the event.
Published Time: 28.11.2023 20:32:07
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: Our investigation into the authentication problem that may be contributing to impact is ongoing as we review the configuration of the affected infrastructure to cultivate our mitigating actions.
Scope of impact: Some users expecting to receive email in any Exchange Online connection method are impacted.
Root cause: A subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact.
Next update by: Wednesday, November 29, 2023, at 6:00 AM UTC
Published Time: 28.11.2023 17:22:52
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: Due to an unexpected conflict, we were unable to move email traffic away from this unhealthy set of infrastructure. We reviewed the network logs to identify an alternate solution to remediate impact and discovered an authentication problem that may be contributing to this issue. We're investigating the configuration of the affected infrastructure to better understand this authentication issue, potentially allowing us to create a strategy that remediates impact.
Scope of impact: Some users expecting to receive email in any Exchange Online connection method are impacted.
Root cause: A subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact.
Next update by: Wednesday, November 29, 2023, at 1:00 AM UTC
Published Time: 28.11.2023 15:31:07
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: We've identified that a subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact. We're moving email traffic away from this unhealthy set of infrastructure to remediate impact.
Scope of impact: Some users expecting to receive email in any Exchange Online connection method are impacted.
Root cause: A subset of infrastructure that routes the affected email isn't processing data as expected, resulting in impact.
Next update by: Tuesday, November 28, 2023, at 10:30 PM UTC
Published Time: 28.11.2023 14:59:47
Title: Some users may not be receiving email messages in all Exchange Online connection methods
User impact: Users may not be receiving email messages in all Exchange Online connection methods.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.
The short version: After checking the configuration of the affected infrastructure, in which emails were not being forwarded, Microsoft technicians developed a fix. This ensured that the affected part of the infrastructure, which is supposed to forward the affected email messages, was replaced. Telemetry has confirmed that the impact has been resolved.
