Why Was P&H Down, and what now?

[pimp]

Hello Guys, as you know we came back from almost 8 days of downtime.

Here is a brief overview of the events as they transpired. Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on. However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol.

The data center experienced some damage. Customer’s servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers. The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt. As of now, we are working to restore power, network, and unaffected hardware back up online within the next day or two.

We thank you for your continued support and understanding. We try and plan for worst-case scenarios and hope for the best in all that we do. We ask that you please submit only one support ticket and refrain from opening multiple support requests at this time until the bulk of the situation is resolved. Please understand that we are working feverishly to resolve this as fast as possible.


After this one of the main servers and 1 of the data servers has been resorted, and few other servers have hardware issues, the problem is there is a big queue of customers like us who are waiting on line to have their servers fixed/rebuilt.

Once this happens and they fixed our servers, all of your data should be back online and running. so please hold with us as we are doing all we can to push on the DC to get our stuff back ASAP.

Thanks for understanding & your patience.

[jeret3225jh]

unable to empty trash. the files just re appear.
Is that something you are working on?

[Slut-lover]

all the files before 2018 seem to be missing
anyone else?

[slimjim]

Yes, it seems that the older files are on servers that are still to be rebuilt. What I have noticed is that in these albums although the pics are not visible on screen if I put my cursor on the blank spaces the data info on the pic does appear so I'm confident the pics are not lost - it's just a matter of time until the servers get fixed.

[Slut-lover]

okay yeah seen that on some but not all

[pimp]

Yeh Guys, Trashbin is disabled for now, we disabled it to not overload the servers that work, all other images (older ones) will be fixed once hosting fixes our delivery servers, so don't worry about it, its only matter of time. we really hope and hope to get it restored. the only problem is we have no clear ETA, they told us between "days to weeks" which is the shittiest reply ever, but they have hundreds if not thousands of servers in need of manual restore, we push as much as possible to get it fixed ASAP.

Do not worry, we are on top of it all 24/7,and once we have more news we will update this thread.

[pimp]

Update:

One of our servers came back, and we are fixing it up/backing it up now, once its done we will need to replace 2 HDs that are faulty there, and then rebuild raid, once fully fixed we will live it and it will start to restore missing images. (i hope within like 48~hours)

Also:

We will need to take the site down to replace few HDs on few of the servers, just letting you know to prepare u`ll see some downtime before we are fully back.

[vampiredic2020]

Thanks for the update!

[pimp]

Update: 1 of 2 broken hard disks has been replaced and now the machine is rebuilding it from raid we are currently at 22%, its doing around 2-3%/hour, once its done, we will take the site down for another 10min or so, while the second HD going to be replaced and another 1.5 days will be spent on rebuilding it, once this is done the main machine will be fully up/stable.

and we will start replacing the secondary servers HDs, once done, we will be online back with all the content. (hopefully no more surprises.)

stay tuned, we are here with you

[slimjim]

Cheers for the update Pimp.