Planned Maintenance In Progress

Updated a few seconds ago

Description

No impact expected. CoreWeave's Storage team are preparing planned maintenance for the week of 2024/02/19, in conjunction with our storage vendor. This maintenance is in response to reductions in SharedFS service quality on 2024/02/08 & 2024/02/09.

Components

CoreWeave Object Storage

Locations

US-ORD1, US-LAS1, US-LGA1

Schedule

February 20, 2024 2:00PM - 9:00PM UTC



February 20, 2024 3:15PM UTC
[Update] No impact expected. CoreWeave's Storage team are preparing planned maintenance for the week of 2024/02/19, in conjunction with our storage vendor. This maintenance is in response to reductions in SharedFS service quality on 2024/02/08 & 2024/02/09.

February 22, 2024 8:06PM UTC
[Update] Maintenance for LAS1 and LGA1 block and sharedfs storage classes are complete. ORD1 maint is being postponed to Feb 26 1800UTC.

CoreWeave Cloud




Operational

CoreWeave Inference




Operational

Concierge Render




Operational

CoreWeave Shared File System Volumes




Operational

CoreWeave Object Storage




Operational

CoreWeave Block Storage Volumes




Operational

CoreWeave Services




Operational

1

Upcoming Maintenances

15

Incidents Last 30 Days

3

Maintenances Last 30 Days

External Services

Scheduled Maintenance

Schedule

February 26, 2024 6:00PM - February 27, 2024 1:00AM UTC

Components

CoreWeave Shared File System Volumes, CoreWeave Object Storage, CoreWeave Block Storage Volumes

Locations

US-ORD1

Description

No impact expected. CoreWeave's Storage team are preparing planned maintenance in ORD1 in conjunction with our storage vendor. This maintenance is in response to reductions in SharedFS service quality on 2024/02/08 & 2024/02/09.

History (Last 7 days)

Degraded Metrics PerformanceDegraded Performance

Incident Status

Degraded Performance


Components

CoreWeave Services


Locations

US-ORD1, US-LAS1, US-LGA1




February 23, 2024 12:24PM UTC
[Investigating] We are actively investigating a metrics ingestion issue that may be impacting the availability of our metrics, we expect the service to come back up. As the situation evolves, we will continue to provide our customers with updates on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 23, 2024 1:02PM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 23, 2024 2:57PM UTC
[Investigating] We are actively investigating a metrics ingestion issue that may be impacting the availability of our metrics, we expect the service to come back up. As the situation evolves, we will continue to provide our customers with updates on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 23, 2024 6:50PM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 23, 2024 9:27PM UTC
[Resolved] We believe this issue to be resolved and in a stable state.

Description

No impact expected. CoreWeave's Storage team are preparing planned maintenance for the week of 2024/02/12, in conjunction with our storage vendor. This maintenance is in response to reductions in SharedFS service quality on 2024/02/08 & 2024/02/09.


Components

CoreWeave Shared File System Volumes, CoreWeave Object Storage, CoreWeave Block Storage Volumes


Locations

US-ORD1, US-LAS1, US-LGA1


Schedule

February 13, 2024 2:00PM - February 16, 2024 8:00PM UTC



February 13, 2024 2:00PM UTC
[Update] Scheduled maintenance is starting.

February 23, 2024 9:00PM UTC
[Update] Scheduled maintenance is complete.
Storage IncidentDegraded Performance

Incident Status

Degraded Performance


Components

CoreWeave Shared File System Volumes


Locations

US-LGA1




February 23, 2024 3:29PM UTC
[Investigating] Hello, We are actively investigating a storage issue that may be impacting our storage services. As the situation evolves, we will continue to update our customers on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 23, 2024 4:10PM UTC
[Monitoring] We believe the issue has been mitigated and we are actively monitoring the situation. We are working on mitigating the underlying root cause to prevent this from happening again. If you are still experiencing an issue please reach out to [email protected].

February 23, 2024 5:59PM UTC
[Resolved] We believe this issue to be resolved and in a stable state.

Description

Please be advised on Friday February 23nd 2024 at 4am UTC(Thursday, February 22st 11p ET) We will be rolling out a reliability improvement change to our DNS infrastructure. Workloads that rely on nginx or similar software that maintain connection sockets to DNS resolvers may need to be restarted at completion of the window if they continue to experience resolution failures. We expect this maintenance window to last approximately two hours, you can expect to see a gap in metrics during this time. Please know this information will backfill after maintenance has been completed. Thank you for your understanding, if you have any questions reach out to [email protected].


Components

CoreWeave Cloud, CoreWeave Inference, Concierge Render, CoreWeave Shared File System Volumes, CoreWeave Object Storage, CoreWeave Block Storage Volumes, CoreWeave Services


Locations

US-ORD1, US-LAS1, US-LGA1


Schedule

February 23, 2024 4:00AM - February 23, 2024 6:00AM UTC



February 23, 2024 4:02AM UTC
[Update] CoreWeave has begun maintenance.

February 23, 2024 4:42AM UTC
[Update] DNS Infrastructure maintenance complete.
Degraded Metrics PerformanceDegraded Performance

Incident Status

Degraded Performance


Components

CoreWeave Services


Locations

US-ORD1, US-LAS1, US-LGA1




February 22, 2024 11:14AM UTC
[Investigating] We are actively investigating a metrics ingestion issue that may be impacting the availability of our metrics, we expect the service to come back up. As the situation evolves, we will continue to provide our customers with updates on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 22, 2024 12:17PM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 22, 2024 3:14PM UTC
[Investigating] [Investigating] We are actively investigating a metrics ingestion issue that may be impacting the availability of our metrics, we expect the service to come back up. As the situation evolves, we will continue to provide our customers with updates on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 22, 2024 4:20PM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 22, 2024 8:22PM UTC
[Resolved] We believe this issue to be resolved and in a stable state.

Incident Status

Service Disruption


Components

CoreWeave Services


Locations

US-ORD1, US-LAS1, US-LGA1




February 22, 2024 3:25AM UTC
[Identified] We are actively investigating a metrics ingestion issue that may be impacting the availability of our metrics, we expect the service to come back up. As the situation evolves, we will continue to provide our customers with updates on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 22, 2024 5:33AM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 22, 2024 6:10AM UTC
[Resolved] We believe the issue to be resolved and in a stable state.
Degraded Metrics PerformanceDegraded Performance

Incident Status

Degraded Performance


Components

CoreWeave Services


Locations

US-ORD1, US-LAS1, US-LGA1




February 19, 2024 12:44AM UTC
[Investigating] Hello, We are actively investigating a metrics issue that may be impacting the availability of our metrics. As the situation evolves, we will continue to provide our customers with updates on the matter. If you have any questions, we encourage you to reach out to support by email: [[email protected]](mailto:[email protected]) We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 19, 2024 2:37AM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 19, 2024 6:19AM UTC
[Resolved] We believe the issue to be resolved and in a stable state.

February 19, 2024 2:33PM UTC
[Monitoring] We believe the metrics issue has been mitigated but we are continuing to actively monitor the affected services. If you are experiencing an issue please reach out to [email protected].

February 19, 2024 3:28PM UTC
[Resolved] We believe the issue to be resolved and in a stable state.
Storage IncidentDegraded Performance

Incident Status

Degraded Performance


Components

CoreWeave Shared File System Volumes


Locations

US-LGA1




February 16, 2024 7:34PM UTC
[Investigating] Hello, We are actively investigating a storage issue that may be impacting our storage services. As the situation evolves, we will continue to update our customers on the matter. If you have any questions, we encourage you to reach out to support by email: [email protected] We apologize for any inconvenience this may cause and thank you for your patience while we work to resolve the matter in a timely manner. Thank you, CoreWeave Support

February 16, 2024 9:13PM UTC
[Monitoring] We've believe the issue has been mitigated and we are actively monitoring the situation. The cause of the issue unexpected error condition that occurred during our original storage maintenance window which required our team to take additional steps to mitigate. If you are still experiencing an issue please reach out to [email protected].

February 16, 2024 10:11PM UTC
[Resolved] We believe the issue to be resolved and in a stable state.