Active Incident

Updated a few seconds ago

Degraded CoreWeave CKS ServicesOperational

Incident Status

Operational

Components

CoreWeave Services

Locations

US-EAST-04


October 13, 2024 3:36PM UTC
[Investigating] We are currently investigating CKS issues in US-EAST-04, which is experiencing disruptions. We are actively working on this and will continue to provide updates as more information becomes available. If you have any questions, please contact support via email.[support@coreweave.com] We apologize for any inconvenience this may cause and appreciate your patience as we address the matter.

October 13, 2024 5:18PM UTC
[Monitoring] We have implemented measures to mitigate the CKS issues in US-EAST-04. It is now operational. We are continuing to investigate and monitor it to ensure stability. Thank you for your patience, and we apologize for any inconvenience caused.

CoreWeave Cloud




Operational

CoreWeave Inference




Operational

CoreWeave Shared Filesystem Volumes




Operational

CoreWeave Object Storage




Operational

CoreWeave Block Storage Volumes




Operational

CoreWeave Services




Operational

CoreWeave Premium Shared Filesystem Volumes




Operational

CoreWeave Logging




Operational

1

Upcoming Maintenances

11

Incidents Last 30 Days

8

Maintenances Last 30 Days

External Services

Cloudflare

Docker

Scheduled Maintenance

Scheduled CoreWeave Logging Maintenance - All regionsPlanned Maintenance

Schedule

October 15, 2024 3:00PM - 9:00PM UTC

Components

CoreWeave Logging

Locations

US-ORD1, US-EWR1, US-LAS1, US-LGA1, US-RNO2, US-EAST-02, US-EAST-04

Description

Details: CoreWeave will be performing an upgrade and maintenance Tuesday, October 15 from 3:00 PM UTC to 9:00 PM UTC that will affect our logging services. Location: All regions. Scope: Upgrades Loki and Promtail to the latest version 3.2.0. Impact: All components will be synced in a rolling fashion to avoid impact. However, if we experience any problems with the upgrade, there is a small chance that a logging service degradation or latency could occur. We will be actively monitoring the situation during this time. We appreciate your patience and understanding. If you have any further questions or require assistance, please don't hesitate to reach out to our support team at support@coreweave.com This activity is deemed Planned Maintenance under the CoreWeave Terms of Service Maintenance Policy ( https://docs.coreweave.com/policies/terms-of-service/maintenance-policy )

History (Last 7 days)

Degraded CoreWeave CKS ServicesDegraded Performance

Incident Status

Degraded Performance


Components

CoreWeave Services


Locations

US-EAST-04



October 13, 2024 1:50PM UTC
[Investigating] We are currently investigating CKS issues in US-EAST-04, which is experiencing disruptions. We are actively working on this and will continue to provide updates as more information becomes available. If you have any questions, please contact support via email.[support@coreweave.com] We apologize for any inconvenience this may cause and appreciate your patience as we address the matter.

October 13, 2024 2:18PM UTC
[Resolved] We believe the issue is resolved now. Our services in US-EAST-04 remain stable and operational. If you are continuing to experience issues, please reach out to support@coreweave.com.
Degraded CoreWeave CKS ServicesPartial Service Disruption

Incident Status

Partial Service Disruption


Components

CoreWeave Services


Locations

US-EAST-04



October 12, 2024 8:52PM UTC
[Investigating] We are currently investigating CKS issues in US-EAST-04, which is experiencing disruptions. We are actively working on this and will continue to provide updates as more information becomes available. If you have any questions, please contact support via email.[support@coreweave.com] We apologize for any inconvenience this may cause and appreciate your patience as we address the matter.

October 12, 2024 9:27PM UTC
[Monitoring] We have implemented measures to address the CKS issues in US-EAST-04 and are currently monitoring it to ensure stability. Thank you for your patience, and we apologize for any inconvenience caused.

October 12, 2024 10:20PM UTC
[Resolved] We believe the issue is resolved now. Our services in US-EAST-04 remain stable and operational. If you are continuing to experience issues, please reach out to support@coreweave.com.
Network Incident Partial Service Disruption

Incident Status

Partial Service Disruption


Components

CoreWeave Services


Locations

US-EAST-04



October 10, 2024 6:11PM UTC
[Identified] We are currently investigating intermittent networking issues in US-EAST-04, where some traffic to the public internet is experiencing disruptions. This may result in intermittent packet loss and connection drops for some services. We are actively working on this and will continue to provide updates as more information becomes available. If you have any questions, please contact support via email.[support@coreweave.com] We apologize for any inconvenience this may cause and appreciate your patience as we address the matter.

October 10, 2024 6:59PM UTC
[Monitoring] We have implemented measures to address the issue and are currently monitoring the network to ensure stability. Thank you for your patience, and we apologize for any inconvenience caused.

October 10, 2024 7:42PM UTC
[Resolved] We have made adjustments and are seeing improvements in the network. If you experience any further issues, please don't hesitate to reach out to us.[support@coreweave.com]
Scheduled CoreWeave Metrics Maintenance - LAS1, LGA1, RNO2Planned Maintenance

Description

Details: CoreWeave will be performing metrics maintenance on Thursday, October 3, 2024 from 3:30 PM UTC to 4:30 PM UTC. Location: LAS1, LGA1, RNO2 Scope: We will be updating the job label on kube-apiserver metrics from job=kube-apiserver to job=apiserver for legacy (non-CKS) deployed control planes. Impact: Non-mitigated alerts and dashboards that are keying off of job="kube-apiserver" will stop working. Any queries, alerts, or dashboards using job="kube-apiserver" should be changed to use job=~"(kube-)?apiserver" or equivalent form of expression for the desired metrics. We will be actively monitoring the situation during this time. We appreciate your patience and understanding. If you have any further questions or require assistance, please don't hesitate to reach out to our support team at support@coreweave.com This activity is deemed Planned Maintenance under the CoreWeave Terms of Service Maintenance Policy ( https://docs.coreweave.com/policies/terms-of-service/maintenance-policy )


Components

CoreWeave Services


Locations

US-LAS1, US-LGA1, US-RNO2


Schedule

October 3, 2024 3:30PM - October 3, 2024 4:30PM UTC


October 9, 2024 7:14PM UTC
[Update] Maintenances was previously completed.
Emergency CoreWeave Storage Maintenance - ORD1 Planned Maintenance

Description

Details: CoreWeave will continue a previous maintenance on Wednesday, October 9, from 2:10 PM UTC to 3:10 PM UTC, affecting a Ceph Storage Disk. Location: ORD1 Scope: Replacing one disk that has been identified at risk of failure. Impact: While the node is taken offline, there is a small risk of connectivity impact as the failover happens, especially for the Filesystem service. We will be actively monitoring the situation during this time. We appreciate your patience and understanding. If you have any further questions or require assistance, please don't hesitate to reach out to our support team at support@coreweave.com This activity is deemed Planned Maintenance under the CoreWeave Terms of Service Maintenance Policy ( https://docs.coreweave.com/policies/terms-of-service/maintenance-policy )


Components

CoreWeave Shared Filesystem Volumes, CoreWeave Object Storage, CoreWeave Block Storage Volumes


Locations

US-ORD1


Schedule

October 9, 2024 2:10PM - October 9, 2024 3:10PM UTC


October 9, 2024 2:08PM UTC
[Update] Urgent maintenance has started.

October 9, 2024 2:54PM UTC
[Update] Urgent maintenance has finished.
Emergency CoreWeave Storage Maintenance - ORD1Planned Maintenance

Description

Details: CoreWeave will be performing a maintenance on Tuesday, October 8, from 8:30 PM UTC to Wednesday, October 9 at 12:00 AM UTC that will affect our Ceph Storage Disks. Location: ORD1 Scope: Replacing 4x disks that have been identified at risk of failure. Impact: While some of the nodes are taken offline, there is a small risk of connectivity impact as the failover happens, especially for the Filesystem service. We will be actively monitoring the situation during this time. We appreciate your patience and understanding. If you have any further questions or require assistance, please don't hesitate to reach out to our support team at support@coreweave.com This activity is deemed Planned Maintenance under the CoreWeave Terms of Service Maintenance Policy ( https://docs.coreweave.com/policies/terms-of-service/maintenance-policy )


Components

CoreWeave Shared Filesystem Volumes, CoreWeave Object Storage, CoreWeave Block Storage Volumes, CoreWeave Services


Locations

US-ORD1


Schedule

October 8, 2024 8:30PM - October 9, 2024 12:00AM UTC


October 8, 2024 8:58PM UTC
[Update] Planned maintenance has started.

October 9, 2024 12:22AM UTC
[Update] This maintenance is being extended for 2 hours past the original scheduled maintenance window. The maintenance is expected to end by 2:00 UTC

October 9, 2024 2:30AM UTC
[Update] Emergency maintenance has been complete.