tag:status.poniverse.io,2005:/historyPoniverse Status - Incident History2024-03-18T19:30:30-07:00Poniversetag:status.poniverse.io,2005:Incident/176926852023-06-27T13:04:40-07:002023-06-27T13:04:42-07:00MLP Forums software upgrade<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>13:04</var> PDT</small><br><strong>Completed</strong> - All done! The upgrade was nice and uneventful (as it should be) but if you run into any weirdness, please report it here: https://mlpforums.com/forum/378-technical-support-desk/</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>14:54</var> PDT</small><br><strong>Scheduled</strong> - We'll be upgrading the core backend software of MLP Forums. The actual downtime is expected to be several minutes but we're setting a generous time window aside just in case.</p>tag:status.poniverse.io,2005:Incident/175644672023-06-26T14:49:26-07:002023-06-26T14:49:28-07:00MLP Forums outage<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>14:49</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:22</var> PDT</small><br><strong>Monitoring</strong> - MLP Forums is back online. Our registrar accepted our explanation and unsuspended the domain though we still have some following-up to do.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>13:28</var> PDT</small><br><strong>Identified</strong> - MLP Forums is down because our registrar suspended the mlpforums.com domain. They sent their intent to do so to an unmonitored email address not associated with the domain, meaning we didn't have a chance to discuss it or explore other options before they enacted the suspension. We believe the rationale they provided to be erroneous and informed them of why, and are awaiting their response.<br /><br />Additionally, we also requested an explanation of why they didn't send their intent-to-suspend to any of the domain's designated contact methods which exist for the purpose of receiving such communications. This would likely not have happened if they did so.</p>tag:status.poniverse.io,2005:Incident/174574842023-06-04T11:14:08-07:002023-06-04T11:14:10-07:00Host Intervention - SSD Failure Replacement<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>11:14</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>15:42</var> PDT</small><br><strong>Monitoring</strong> - The first SSD has been replaced and we have gotten restored the status of the drive pool, but the second SSD needs to be replaced. We will schedule this ASAP and update here once we know when it is happening.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>14:24</var> PDT</small><br><strong>Identified</strong> - Services could be intermittently offline throughout the night as we perform emergency maintenance to replace an SSD.</p>tag:status.poniverse.io,2005:Incident/172977792023-05-18T23:47:13-07:002023-05-18T23:49:01-07:00Poniverse-wide outage<p><small>May <var data-var='date'>18</var>, <var data-var='time'>23:47</var> PDT</small><br><strong>Resolved</strong> - The root cause of today's outage was identified and fixed. The same issue won't cause a failure again.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>23:28</var> PDT</small><br><strong>Monitoring</strong> - Issue was identified and corrected. How it ended up in this state is being investigated for possible future preventive actions but we're back online!</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>22:12</var> PDT</small><br><strong>Investigating</strong> - Our host got back to us and is investigating the issue with us.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Identified</strong> - There's a problem causing all our services to be unreachable. Our IP addresses are not routing to our server and signs point toward this being an issue with our host's network. We have contacted them to investigate and are awaiting their response.</p>tag:status.poniverse.io,2005:Incident/112905182023-02-26T00:36:11-08:002023-02-26T00:36:12-08:002022 Server Migration<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>00:36</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>13:34</var> PDT</small><br><strong>Scheduled</strong> - We will be migrating all of our major services to the new server cluster. We attempted to migrate previously but as there were networking issues with the vRack setup we are going for a full outage instead to get it all done in one go. We should be mostly done by the end of tonight.</p>tag:status.poniverse.io,2005:Incident/112904972023-02-26T00:34:44-08:002023-02-26T00:34:45-08:00Pony.fm Migration Related Outage<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>00:34</var> PST</small><br><strong>Resolved</strong> - We finally managed to get the upload pipeline working again!</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>13:32</var> PDT</small><br><strong>Identified</strong> - As we migrate servers some of our services will be down until we are done. Pony.fm's data and services are migrated to the new host but the database it relies on is still on the old cluster. Until that gets moved with the rest of the data tonight, it will remain down.</p>tag:status.poniverse.io,2005:Incident/112292062022-09-22T22:49:10-07:002022-09-22T22:49:21-07:00Pony.fm maintenance<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>22:49</var> PDT</small><br><strong>Resolved</strong> - The update has been completed and Pony.fm is back online! Among various fixes, Pony.fm now supports native media notifications that let you control the site's playback through your operating system and Bluetooth devices! Thanks to Comp for this particular code contribution. /)</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>21:39</var> PDT</small><br><strong>Identified</strong> - Pony.fm is being updated. Expect it to be offline for some amount of time until the maintenance is completed.</p>tag:status.poniverse.io,2005:Incident/94380222022-03-03T06:48:38-08:002022-03-03T17:01:47-08:00Pony.fm domain expired (accidentally)<p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>06:48</var> PST</small><br><strong>Resolved</strong> - We got the domain back! https://pony.fm/ is up and running again as of 06:48 Pacific Time today. Please return to your regularly scheduled sharing & enjoyment of pony tunes.<br /><br />Shoutout to our registrar and the .fm registry for their understanding & cooperation. Once again, we've taken preventive action to not let the domain surprise-expire on us like this again.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>03:40</var> PST</small><br><strong>Identified</strong> - Pony.fm is currently unavailable because of an off-by-one error, combined with "failsafe" measures that failed, that culminated in unwittingly allowing the domain to expire on us. Whoops.<br /><br />Feeling a lot of sympathy for season 2 Twilight here: https://www.youtube.com/watch?v=Ef50uHGzg0E<br /><br />We're in the process of getting the domain back from the registry - the registrar is cooperating with us but this process takes several business days.<br /><br />Preventive action has already been taken to prevent a recurrence of this in the future. The factors that led to missing pony.fm's renewal are unique to that domain - they do not affect Poniverse's other domains.</p>tag:status.poniverse.io,2005:Incident/65225942021-03-13T11:10:30-08:002021-03-13T11:10:32-08:00Mail server migration<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>11:10</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>09:11</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>09:09</var> PST</small><br><strong>Scheduled</strong> - We're migrating our mail server to a some new shiny hosts. This may cause intermittent problems on our services.</p>tag:status.poniverse.io,2005:Incident/59234632021-01-01T20:52:51-08:002021-01-01T20:53:18-08:00MLPF is going offline for an upgrade!<p><small>Jan <var data-var='date'> 1</var>, <var data-var='time'>20:52</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 1</var>, <var data-var='time'>20:52</var> PST</small><br><strong>Update</strong> - Gonna mark this as closed now, forums are up again.</p><p><small>Dec <var data-var='date'>31</var>, <var data-var='time'>20:10</var> PST</small><br><strong>Update</strong> - Work on the update continues. We apologise for the downtime.</p><p><small>Dec <var data-var='date'>31</var>, <var data-var='time'>17:37</var> PST</small><br><strong>Update</strong> - We are currently investigating why our theme is not working.</p><p><small>Dec <var data-var='date'>31</var>, <var data-var='time'>17:22</var> PST</small><br><strong>Identified</strong> - Just working through some permission issues here. Sorry for the delays.</p><p><small>Dec <var data-var='date'>31</var>, <var data-var='time'>14:27</var> PST</small><br><strong>Investigating</strong> - The upgrade will start at 2:35 PM PST (4:35 PM Central), and is expected to take less than half an hour. The forum will be unavailable during that time.</p>tag:status.poniverse.io,2005:Incident/32476392019-11-24T13:53:23-08:002019-11-24T13:53:24-08:00Widespread Poniverse Services Outage<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>13:53</var> PST</small><br><strong>Resolved</strong> - The new drives are all tickety-boo! Happy to call this incident resolved.<br /><br />Thank you again for your understanding, and have a good day everypony!<br />~tinker</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>00:06</var> PST</small><br><strong>Monitoring</strong> - We are happy to report that the hardware fault in the affected node has been identified and resolved. Two internal SSDs were simultaneously malfunctioning, and they have both been replaced without loss of data (thank you ZFS).<br /><br />Unfortunately the load was too much to handle across our remaining servers, and we did have widespread outages until the affected node could be brought back online. Poniverse apologizes for this extended outage and will seek methods to avoid this in the future.<br /><br />All services should now be back online and functioning at 100% capability. We will continue to monitoring the performance of the new SSDs until we are happy that the incident has been fully resolved.<br /><br />Because we believe in deep ends, all Minecraft servers have been restored as well. Enjoy!<br /><br />Thank you for your patience and understanding during the last two days,<br />The Poniverse Crew<br />~tinker</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>00:16</var> PST</small><br><strong>Identified</strong> - Since 12am GMT 22 November, we have experienced performance issues and entire service outages on across our platform. Our technicians have identified the affected node, and migrated services to other working nodes. This will likely affect the performance of core services, and especially MLP Forums and Canterlot Avenue. Additionally, we will be suspending all Minecraft servers until we have the hardware resources to run them smoothly.<br /><br />Work will continue to determine the affected component of the downed node, and it will be repaired or replaced ASAP to bring Poniverse back to regular performance.<br /><br />We apologize for the inconvenience during this period and wish you all a good weekend.<br />~tinker</p>tag:status.poniverse.io,2005:Incident/29658562019-09-05T23:16:59-07:002019-09-05T23:17:00-07:00Poniverse Server Upgrade<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>23:16</var> PDT</small><br><strong>Completed</strong> - Upgrade completed! ~tinker</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>21:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>19:23</var> PDT</small><br><strong>Scheduled</strong> - We will be updating the core systems of our servers to the latest versions.</p>tag:status.poniverse.io,2005:Incident/25667812019-06-19T23:40:34-07:002019-06-19T23:40:34-07:00Forums are down.<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>23:40</var> PDT</small><br><strong>Resolved</strong> - The problem was solved by reinstalling the forum files.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>01:37</var> PDT</small><br><strong>Update</strong> - This will be a major outage from the looks of it, unfortunately, and will take more than a few hours to resolve. The trick we used to get the forums working last time they had an outage left the system in a weird, unstable state from which we ourselves cannot escape.<br /><br />IPS support has been contacted, and we are currently waiting to hear back from them in the morning.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>01:04</var> PDT</small><br><strong>Identified</strong> - During an attempt to install a plugin, the forums have decided to entirely stop working. While the plugin has been removed, the issue persists.</p>tag:status.poniverse.io,2005:Incident/25288232019-06-13T14:25:02-07:002019-06-13T14:25:04-07:00Hard Drive Replacement for Mount Everhoof<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:25</var> PDT</small><br><strong>Resolved</strong> - Resilver complete! We're back at full redundancy! ~tinker</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>18:54</var> PDT</small><br><strong>Monitoring</strong> - The drive has been replaced and the data is being resilvered. All services are back online ~tinker</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>15:48</var> PDT</small><br><strong>Identified</strong> - We received notification from one of our host nodes that one of the storage drives was in the initial stage of failure (Bad sectors) and we are working with our host to have drives replaced.</p>tag:status.poniverse.io,2005:Incident/23710852019-05-05T19:34:45-07:002019-05-05T19:36:17-07:00MLPF down after failed upgrade.<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:34</var> PDT</small><br><strong>Resolved</strong> - MLPForums restored. 👍 It is now scheduled to be upgraded at some point in the coming week or two.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:31</var> PDT</small><br><strong>Update</strong> - Pony.fm restored. 👍</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:26</var> PDT</small><br><strong>Update</strong> - A storage disk has stopped responding, but MLPF has been somewhat restored.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>17:30</var> PDT</small><br><strong>Update</strong> - Invision support has been contacted. We are also considering a roll-back to a recent snapshot. This may take some time, since something seems to have gone seriously wrong.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>17:15</var> PDT</small><br><strong>Update</strong> - An error has occurred during the update procedure.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>17:01</var> PDT</small><br><strong>Identified</strong> - Here we go.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>16:46</var> PDT</small><br><strong>Investigating</strong> - The upgrade will start at 5 PM PST (8 PM Eastern) and is expected to take less than 15 minutes. The forum will be unavailable during that time.</p>tag:status.poniverse.io,2005:Incident/23698912019-05-04T18:37:32-07:002019-05-04T18:37:34-07:00MLPF is going offline for an upgrade!<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>18:37</var> PDT</small><br><strong>Resolved</strong> - The upgrade has been delayed because we need to examine some files which are going to be overwritten.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>18:30</var> PDT</small><br><strong>Identified</strong> - Starting upgrade.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>18:14</var> PDT</small><br><strong>Investigating</strong> - The upgrade will start at 6:30 PM PST (9:30 PM Eastern), and is expected to take about 15 minutes. The forum will be unavailable during that time, and may have a few bugs to work out once it is back up.</p>tag:status.poniverse.io,2005:Incident/23300902019-04-15T21:38:09-07:002019-04-15T21:38:09-07:00MLPF is going offline for an upgrade!<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>21:38</var> PDT</small><br><strong>Resolved</strong> - The forum upgrade has been completed, and important issues fixed.<br />Thank you for your patience.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>21:37</var> PDT</small><br><strong>Monitoring</strong> - A series of fixes have been deployed. Ongoing monitoring for any potentially remaining bugs will contain, but MLP Forums is back to operational status. ~tinker</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>17:35</var> PDT</small><br><strong>Identified</strong> - MLP Forums has been upgraded to 4.4.2. This has broken some of our theming, resulting in error messages and some missing data. Bug reports and status updates are at https://mlpforums.com/topic/183692-of-upgrades-and-bugs/.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>15:12</var> PDT</small><br><strong>Investigating</strong> - The upgrade will start at 3:30 PM PST (6:30 PM Eastern), and is expected to take an hour. The forum will be unavailable during that time, and may have a few bugs to work out once it is back up.</p>tag:status.poniverse.io,2005:Incident/22482392019-03-07T23:34:58-08:002019-03-07T23:34:58-08:00Storage server network interface will not start<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>23:34</var> PST</small><br><strong>Resolved</strong> - Our hosting provider was notified and resolved the issue, sorry about the downtime!</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>19:17</var> PST</small><br><strong>Identified</strong> - We've identified an issue with our storage server's network interface, public requests to content on this server cannot currently be served. Sites using the internal network appear to be unaffected. We will likely need to get in contact with our hosting provider to investigate this issue.</p>tag:status.poniverse.io,2005:Incident/18000292018-07-04T00:53:47-07:002018-07-04T00:53:47-07:00MLPF is going offline for an upgrade!<p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>00:53</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>00:29</var> PDT</small><br><strong>Investigating</strong> - The upgrade will start at 12:45 AM PST, in about 15 minutes, and is expected to take 5 minutes. The forum will be unavailable during that time.</p>tag:status.poniverse.io,2005:Incident/17450162018-05-28T03:37:37-07:002018-05-28T03:37:37-07:00MLPF: Can't Log In<p><small>May <var data-var='date'>28</var>, <var data-var='time'>03:37</var> PDT</small><br><strong>Resolved</strong> - Logging in should work again.<br />Please report any errors related to logging in on https://discord.gg/poniverse.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>02:17</var> PDT</small><br><strong>Update</strong> - It's our problem, apparently. Investigating the "poniverse" app, whatever that is. 😬</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>01:59</var> PDT</small><br><strong>Identified</strong> - An issue with the login page has been identified. (A redirect-related component seems to be having trouble.)<br />Technical support has been contacted. Please stand by...</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>01:43</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>01:42</var> PDT</small><br><strong>Investigating</strong> - The log-in panel is broken, so no one who's not logged in can log in.</p>tag:status.poniverse.io,2005:Incident/17449822018-05-28T01:06:29-07:002018-05-28T01:07:18-07:00MLPF is going offline for an upgrade!<p><small>May <var data-var='date'>28</var>, <var data-var='time'>01:06</var> PDT</small><br><strong>Resolved</strong> - All upgraded! ^_^<br />You may now resume your regularly scheduled posting.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>00:28</var> PDT</small><br><strong>Investigating</strong> - The upgrade will start at 12:45 AM PST (2:45 AM Central), and is expected to take less than half an hour. The forum will be unavailable during that time.</p>tag:status.poniverse.io,2005:Incident/17016952018-04-23T12:24:32-07:002018-04-23T12:24:33-07:00Upgrade MLPF<p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>12:24</var> PDT</small><br><strong>Resolved</strong> - Upgrade complete.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>12:23</var> PDT</small><br><strong>Monitoring</strong> - Upgrade in process.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>12:08</var> PDT</small><br><strong>Investigating</strong> - MLPF is going offline for an upgrade! The upgrade will start at 12:20 PM PST, and is expected to take about 10 minutes. The forum will be unavailable during that time.</p>tag:status.poniverse.io,2005:Incident/16718032018-04-03T17:23:56-07:002018-04-03T17:23:57-07:00MLPF Outage: CDN<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>17:23</var> PDT</small><br><strong>Resolved</strong> - Rebuilt the themes, and shook our collective heads at IP Board for making us perform dark magic on their software.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>16:13</var> PDT</small><br><strong>Identified</strong> - Changing CDN settings has made the site nonfunctional, working on fixing the settings.</p>tag:status.poniverse.io,2005:Incident/16694902018-04-03T16:01:16-07:002018-08-01T12:38:37-07:00Security incident on Canterlot Avenue<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>16:01</var> PDT</small><br><strong>Resolved</strong> - Canterlot Avenue is back online!</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>21:47</var> PDT</small><br><strong>Identified</strong> - We have finished our investigations. Preparations are being made to bring the site back online.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>09:43</var> PDT</small><br><strong>Investigating</strong> - It appears a bad actor has breached Canterlot Avenue. The site will be offline while an investigation into what happened proceeds. There will be an update when it's done.
<br />
<br />~PoniOps team</p>tag:status.poniverse.io,2005:Incident/16688842018-04-02T00:10:51-07:002018-04-02T00:10:51-07:00Forum Upgrade<p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>00:10</var> PDT</small><br><strong>Resolved</strong> - The forums have been successfully upgraded.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:50</var> PDT</small><br><strong>Investigating</strong> - MLPF is going offline for an upgrade! The upgrade will start at midnight, PST, and is expected to take about half an hour. The forum will be unavailable during that time.</p>