Why Was P&H Down, and what now?
 

Hello Guys, as you know we came back from almost 8 days of downtime.

Here is a brief overview of the events as they transpired. Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on. However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol.

The data center experienced some damage. Customer’s servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers. The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt. As of now, we are working to restore power, network, and unaffected hardware back up online within the next day or two.

We thank you for your continued support and understanding. We try and plan for worst-case scenarios and hope for the best in all that we do. We ask that you please submit only one support ticket and refrain from opening multiple support requests at this time until the bulk of the situation is resolved. Please understand that we are working feverishly to resolve this as fast as possible.


After this one of the main servers and 1 of the data servers has been resorted, and few other servers have hardware issues, the problem is there is a big queue of customers like us who are waiting on line to have their servers fixed/rebuilt.

Once this happens and they fixed our servers, all of your data should be back online and running. so please hold with us as we are doing all we can to push on the DC to get our stuff back ASAP.

Thanks for understanding & your patience.

unable to empty trash. the files just re appear.
Is that something you are working on?

all the files before 2018 seem to be missing
anyone else?

Yes, it seems that the older files are on servers that are still to be rebuilt. What I have noticed is that in these albums although the pics are not visible on screen if I put my cursor on the blank spaces the data info on the pic does appear so I'm confident the pics are not lost - it's just a matter of time until the servers get fixed.

okay yeah seen that on some but not all

Yeh Guys, Trashbin is disabled for now, we disabled it to not overload the servers that work, all other images (older ones) will be fixed once hosting fixes our delivery servers, so don't worry about it, its only matter of time. we really hope and hope to get it restored. the only problem is we have no clear ETA, they told us between "days to weeks" which is the shittiest reply ever, but they have hundreds if not thousands of servers in need of manual restore, we push as much as possible to get it fixed ASAP.

Do not worry, we are on top of it all 24/7,and once we have more news we will update this thread.

Update:

One of our servers came back, and we are fixing it up/backing it up now, once its done we will need to replace 2 HDs that are faulty there, and then rebuild raid, once fully fixed we will live it and it will start to restore missing images. (i hope within like 48~hours)

Also:

We will need to take the site down to replace few HDs on few of the servers, just letting you know to prepare u`ll see some downtime before we are fully back.

Thanks for the update!

Update: 1 of 2 broken hard disks has been replaced and now the machine is rebuilding it from raid we are currently at 22%, its doing around 2-3%/hour, once its done, we will take the site down for another 10min or so, while the second HD going to be replaced and another 1.5 days will be spent on rebuilding it, once this is done the main machine will be fully up/stable.

and we will start replacing the secondary servers HDs, once done, we will be online back with all the content. (hopefully no more surprises.)

stay tuned, we are here with you :)

Cheers for the update Pimp. :)

no problem, you know we are here for you all :)

P.S: rebuild at 32%

First HD was rebuilt at 100% and then failed :(

Now HD #2 is rebuilding at 16% progress ... and then we are going to try and rebuild hd #1 yet again.

So finally 1 HD rebuilt O.K, we replaced the second HD which had failed last time, and it`s rebuilding right now.

P.S: One of the servers that is now missing and that's why lots of images are down, is also backing up, i hope in the next few days its DONE backing up and we will start rebuilding HD there also and serving from it.

Hopefully we will be able to restore the site fully soon enough. thanks for your patience boys & girls!

Thanks again for the latest news

So FINALLY the "missing" server that was backing up and it took (9 days) to back it up so we wont have nothing lost in case of it going south.

now its back & HD is being rebuilt and on top of it all its now serving files, so it means all your missing files will start to repopulate since this second :) once its rebuild is done, we will be safe and know all is good and safe, I'll keep updating here.

Well good news at my end. I had a few albums with "missing" pics and they have all been restored.

In addition in 2 albums I had approx. 30No. pics with the "Pimp&Host - this image has been removed" showing (these were old pics uploaded in 2017) but checking a few minutes ago in one album these has now been fully restored too leaving just 10 pics in the 2nd album which I guess will appear before too long.

Uploading seems to be working fine, looks like emptying the trash bin is still disabled - but so what?, a minor detail given where we were.

Looks like you and your techies can stop holding your breathe and chill out! Many thanks guys.

So yeah, the images with "Image Removed" is wrong permissions, we do have those images, and we are fixing the permissions right now, it will take some time though since we have gazillions of images, but don't delete the albums the images will be restored also.

Regarding the trashbin/statistics/archives, those are turned off right now until all our servers will be in action, which might take some more time, but again all those are noted and we are fully aware of everything, all will be restored to fully capability hopefully in the next 1-2 weeks.

Hi Pimp... So the main site is currently down. Is this a planned outage to enable maintenance/server switching etc.? and if so how long do you anticipate being offline for?

..................

edit - it's back up now !!

We had some issue with the Power Supply Unit, and it was replaced now, it seems the servers still suffer from the earlier issues of the data center. hope soon all will be stable.

Don't know. It's been more than 4 weeks that anybody can't download anything.
We can't count on your service, so we'll go elsewhere.
Thank you

"Pimpin is not who you are, it's what you do."

Well this is to bad, as you can see we are here working on restoring everything to full capacity, but you also have to understand our site is HUGE this means tons of servers tons of hds, and restoring takes really long time, but the main functionality & all your images are 100% restored, so yes, zip downloads & cleaning trashbin is halted until we fully restore our servers, if this is something that would make you move, I'm sorry to hear this.