The new drives are all tickety-boo! Happy to call this incident resolved.
Thank you again for your understanding, and have a good day everypony! ~tinker
Posted Nov 24, 2019 - 13:53 PST
Monitoring
We are happy to report that the hardware fault in the affected node has been identified and resolved. Two internal SSDs were simultaneously malfunctioning, and they have both been replaced without loss of data (thank you ZFS).
Unfortunately the load was too much to handle across our remaining servers, and we did have widespread outages until the affected node could be brought back online. Poniverse apologizes for this extended outage and will seek methods to avoid this in the future.
All services should now be back online and functioning at 100% capability. We will continue to monitoring the performance of the new SSDs until we are happy that the incident has been fully resolved.
Because we believe in deep ends, all Minecraft servers have been restored as well. Enjoy!
Thank you for your patience and understanding during the last two days, The Poniverse Crew ~tinker
Posted Nov 24, 2019 - 00:06 PST
Identified
Since 12am GMT 22 November, we have experienced performance issues and entire service outages on across our platform. Our technicians have identified the affected node, and migrated services to other working nodes. This will likely affect the performance of core services, and especially MLP Forums and Canterlot Avenue. Additionally, we will be suspending all Minecraft servers until we have the hardware resources to run them smoothly.
Work will continue to determine the affected component of the downed node, and it will be repaired or replaced ASAP to bring Poniverse back to regular performance.
We apologize for the inconvenience during this period and wish you all a good weekend. ~tinker
Posted Nov 23, 2019 - 00:16 PST
This incident affected: Core Web Apps (Poniverse.net, MLP Forums, Canterlot Avenue, Pony.fm, Equestria.tv), Partner Sites (BronyCAN.ca, Pony Print Project), Ponyville Live! (Website & API, Radio), and PoniArcade (Minecraft servers).