Hello valued customers,
We received reports in the evening of the 11th regarding issues with the /storage partition on our storage VPS. It appears that the partition has become unusable and is operating in read-only mode. This is a familiar error, but the circumstances this time are unusual. Despite the mdadm RAID failing, the partition on the node remains intact, even though no HDDs are defective.
Fortunately, due to the more intricate RAID configuration on STOR-4, we have metadata backups for all VPS on the node from the /storage partition. This means there's no risk of data loss. However, WE REQUEST URGENT/ASAP all customers to promptly back up any important data from /storage partition, as we will need to rebuild the partition. Only the /storage partition will sufered a complete reinstallation, the main partition (/)still be intact of the VPS.
Q: How much time do I have to MAKE BACKUP?
A: We're allow all customers 96 hours (4 days) time to make backup. Additionally, we'll grant a 2-week extension on the next billing date.
Q: What caused this error?
A: We're still investigating the exact cause of this incident. However, it appears that the configuration differences, particularly the buffer-cache setup on STOR-4, may have contributed. Here's the current configuration:
20x 18 TB HDD SAS RAID 60
1x 960 GB SSD SAS for boot
1x 960 GB SAS for mdadm buffer cache
2x 2 TB HDD SAS RAID 1 for mdadm metadata backup
For prevent and improvment node we start make next changes:
We intend to remove the buffer-cache during the reconfiguration of /storage to prevent similar issues in the future.
We start fix I/O speed , we will use the old cache method used on STOR-1 / STOR-2 and STOR-3 which was very efficient, and since we had the nodes up and running in the last 6-7 months everything went perfectly without any data loss
PLEASE DON'T REBOOT/SHUTDOWN YOU VPS FOR DON'T PREVENT A PERMANENTLY DATA LOST
We start post more updates on our discord server > https://discord.gg/7cnRBWEW8u and on our website page > https://panel.ihostart.com/index.php?rp=/announcements
Thanks for understand and sorry for this!
Regards,
Calin - C.E.O iHostART
