What is Digg?11 Comments
- stutter, on 10/12/2007, -0/+2After reading the conversation that had taken place between your staff and David B. I can say that you didn't do anything "very gladly" to assist with this colossal mess up. It took several hours for you to even figure out what actually did happen to the account. Maybe you should take a look at the code behind your automated system, there is obviously a flaw. I wasn't aware it was common practice to delete customers' files in leu of suspending the account per further follow up. If your company is hurting for disk space that much, maybe you should think about increasing your prices in order to fund such an endeavor, of course if you raised your prices you'd probably have to look into seriously enhancing your customer support.
- lob-sogular, on 10/12/2007, -0/+2Two Rules of Software Development:
1) Unless you are 100% sure that your software is 100% correct, do not automate deletion without backing up the data.
2) You are never 100% sure that your software is 100% correct.
I find it hard to believe that any company is having enough delinquent accounts to necessitate automated deletion. And if the company is having enough, then perhaps it should examine why. But perhaps this was just some brilliantly idea which was not thought through: "If we automate the deletions, then that means less work for us!"
Either way, it shows gross incompetence on Steadfast Networks' part. - fac3less, on 10/12/2007, -0/+2I was with steadfast for awhile myself -- they're decent but they lack quite a bit in the support department although mainly only regarding network issues.
http://hostjury.com/reviews/steadfast-networks/ - ajvan, on 10/12/2007, -0/+2Nice that the hosting co. was called Steadfast.
- gizmola, on 10/12/2007, -0/+2We had a VPS site with steadfast as of about 2 hours ago. We received a rash of emails that had been backed up in their billing system, indicating that we were past due on a payment to cover the next 3 months. The guy who pays the site, uses the invoice to make a paypal payment once he gets the invoices, which in this case never came.
Earlier this evening (Saturday btw!) some batch job on their side kicked in, and the site was turned off, and according to tech support deleted. After talking to tech support "probably" all of our data was deleted.
So to be clear, when their broken billing system determined that we were due to pay our next payment about 35 days ago, it allowed for a 2 week grace period. Once it decided that we were past due, it rm -f'd our entire site. I'd also note for the record, that they have a phone number on file that they could have called.
I could go on quite a bit about the number of things that are completely stupid about the way they've set their system up, although at the moment, I'm hoping against hope that the month or so of data in the system (it's a community site, with forums and user contributed content and data that all goes into a database and onto directories on the filesystem that are written by php scripts) isn't completely gone.
I'm not sure that this will kill the site, but it's not looking good. I'll return here and provide an update once I know more. - gizmola, on 10/12/2007, -1/+2Steadfast are tools.
Yup, they killed our site. Same story... a vps that they don't back up. They have a nice story about that that makes no sense. Their software is buggy. Mails backed up, no notice, kablam.... they rm -rf our site, a 3 year old free movie review site. I have the screen shots of our account showing that it indicated no payment problem. Yes, the dumb kid who handled the finance side of the site, didn't monitor the payment status (we were paying quarterly by paypal) and we owed a payment, but Steadfasts email engine was clearly backed up, because we got 4 emails all in one shot, and this was on a Friday night, whenever their billing cron job runs.
Clearly Steadfast is such a moronic fly by night crap works, that reclaiming the same amount of diskspace Google gmail gives away free is more important to them than a customer's site. It's not a case of not having a backup, it's a case of someone you are paying for a service, who, the hour their system shows you owe them a dollar, automatically goes and wipes the disk clean. That's just bad business practice designed by someone who's clueless and should get out of the hosting business.
Did we have a backup? Yes.. of course our site was free as well, so we weren't as current as we should have been and lost several weeks of reviews, and forum contents. It was enough of a loss for us that it killed the site.
Steadfast's attitude during the entire thing was completely assinine. Not only did they tell us it was our fault, they never offered one bit of help, or a dollar credit. - garrett, on 10/12/2007, -1/+2This is seriously screwed up...
- justinshattuck, on 10/12/2007, -1/+2I agree garrett, ***** hosers.
- kstange, on 10/12/2007, -2/+0It is true we made no attempt to recover the data. It had most definitely already been overwritten by other customer data and it was not possible to do such recovery without shutting down over 30 other customers' VPS servers. We regret that the billing issue occured that resulted in the account being deleted, but there was nothing we could do after it happened to recover the deleted content. It was an automated system that deleted the account, not anything malicious on our part, and we would certainly like to make sure that is understood by anyone wishing to hold a grudge. If we could have recovered the files, we would have very gladly have retreived and returned them to the site owner.
- kstange, on 10/12/2007, -2/+0The root of the problems seem to have been related to the fact that the customer account was not suspended at the time it was being deleted. Our logs say that it was stopped on schedule, but it seems that this either did not happen for some reason or someone restarted the account accidentally without remembering to check that it was past due. In any case, the system (to some customers' annoyance) dispatches an absurd number of warning emails for virtually ever automated action imaginable and one (or many more) of those should have been received. We can't really take responsibility for emails we sent that were not received.
I believe the delay in determining what had happened was because the tech on duty was not familiar with the billing system (as that is not his job) and myself and one other staff member were contacted to try to determine why the account was marked for and subsequently deleted. According to the ticket he submitted, it took approximately 45 minutes for us to determine what happened which included the tech's investigation, then him containing two other staff members, and our investigation, and reply time. I don't know where the "several hours" come from. The end result is that the account was deleted because the customer did not pay, and the deletion is permanent.
It is important for customers to remember to pay their bills on time. We allow servers to operate for several weeks for free before shutting it down, then leave them for several more weeks offline before deletion. We give that much time so that customers have the chance to try to resolve any billing issues, request extensions due to unusual financial circumstances or bring discrepancies to our attention. The customer chose to pay with a manual method, so it was up to him to make sure to send the payments every month. As far as I can tell, the regularly monthly payments were not submitted.
I don't see any historical information with this customer that suggests we did anything other than try to explain what happened. It's very sad that he had just deleted his backups before this incident occurred, but that's not anything we can do anything about. He was informed that data recovery was not possible, which was true, so the only remaining thing we could do was try to determine what occurred and inform him. That seems to have been done. - kstange, on 10/12/2007, -2/+0Just for the record, I am not going to hang around any post any more comments here. I just wanted to make sure to clarify for anyone reading that what happened was not a malicious act and that we did all we could do after the fact. We regret that we are unable to help this customer further in recovering his lost content, which it seems is the result of many, many hours of hard work. I hope that he is able to recover or rebuild something and bring back to his loyal users the services lost and that such a disaster does not happen again wherever he chooses to host it.
We are going to endeavor to more carefully verify the billing details of people paying manually to ensure this sort of problem does not repeat itself. This is not something we guarantee, but it is the least we can do to come away from this incident with a preventative policy adjustment.
I do not anticipate that we will win anyone reading this story as customers, and that is understandable, but thank you for reading.
Check out the new & improved 
© Digg Inc. 2009
— Content created and posted by Digg users is