posted on February 20, 2006 06:53:09 AM new
I would like some answers from Vendio on a few issues that came up during the outage. I am not going to beat them up any more, that will not move us forward, but will make us feel a little better .
In the event of another outage such as this - how do we reach you to find out what is going on?
The system status was showing as working optimally for nearly 1 hour AFTER the outage began. Why was this not updated sooner? People were extremely upset that they did not know if you even KNEW there was an outage. How did you find out about it?
Several of us were using your 'contact us' form with outage as the subject - is that how you were alerted to the issue?
Is there a way to flash an alert across all the forums regarding outages and status? Many people, like myself, tend to hang out on one board (Ebay outlook for me) and may not check all the others?
Please start a dialog with us about this, it was pretty frustrating and costly for many of us.
~~~~~~~~~~~~~~~~~~ Caroline
posted on February 20, 2006 08:39:51 AM new
Yes, please answer discuss this with us, your loyal customers! In addition to Caroline's questions, please discuss these topics I've posted already (and haven't received a response from):
1) Why did it happen? What caused this extensive outage? (I know you say it was a corrupted server... but as an IT professional, that's not nearly a good enough answer. It's the ISP version of a computer repair person saying... "well, I guess your computer was hit by a virus" (when he really has no clue what happened). We'd kind of like some details... be as technical as you can. There are plenty of IT folks here on the boards that can help explain it to the rest of the people)
2) What measures are you implementing to ensure that it never happens again?
3) What compensation will you be providing us for loss of sales? THIS ONE IS IMPORTANT AND NEEDS TO BE DISCUSSED!!!!
posted on February 22, 2006 10:18:17 AM new
Hello,
Thank you for posting.
I can assure you that we always know about an outage, however, generally it is the engineering staff who first becomes aware of the issue. An engineer who gets an alarm is going to react by immediately focusing on the problem to get it resolved as quickly as possible. This can result in a slight delay in the information getting back to customer service so that we can post an announcement and address message center threads and emails.
We do apologize for any delay in our messaging. We understand your concerns, and we do our best to address issues in a timely way. Admittedly, in this case the delay in getting information through to customer service was longer than normal and we will look into ways to avoid this in the future. I'm sure that you agree, however, that in the event of an outage it is important for our engineers to focus on getting the problem fixed rather than spend energy and time trying to explain what is going on.
I would also like to point out that our image hosting service is extremely reliable. We very rarely experience outages and if we do they are generally short in duration.
posted on February 23, 2006 06:16:25 AM new
Diana,
Thank you for posting a reply. Frankly, however... I am starting to feel ignored. This is the second time I've posted my same exact list of three questions. This is also the second time you've responded to a thread and have neglected to answer any of my questions. Why? I'm also not sure that you really answered all of Caroline's questions either like:
In the event of another outage such as this - how do we reach you to find out what is going on?
Several of us were using your 'contact us' form with outage as the subject - is that how you were alerted to the issue?
Is there a way to flash an alert across all the forums regarding outages and status?
No one is disputing that Vendio typically offers good service. But, these are important questions that need to be answered in order to restore confidence in your relationship with your customers.
One of your posts during the outage indicated that the time for discussing these issues was not during the outage (and I agree) but after the outage had been fixed. I am assuming the problem is fixed, isn't it? (please let us know if you're still experiencing residual effects!) If so... let's start some open communication!
posted on February 23, 2006 11:13:19 AM new
Dear eauctionmanagement,
I apologize for making you feel ignored. I thought I had addressed your questions already, but I will try to formulate my answer in a more specific format.
1. The best way to reach us during an outage is via email, or posting in the message center.
2. No, we were not alerted to the issue by incoming email. Our engineering group was alerted to the issue, and it simply took some time for them to get the message to customer service.
3. In the event of an outage we will post an announcement as quickly as we can (ie. once we have confirmed the outage and have some information from engineering).
4. We restored the images from a backup. The only residual effect that I am aware of is that some images that were uploaded just before the outage started were not able to be restored. The backup was made just before the outage started, so this is a corner case only.
posted on February 23, 2006 11:18:28 AM new
Diana,
Thank you for posting again. I think you've answered Caroline's questions fairly well now, and I appreciate that. However, I'd still like answers to my original three questions (that I posted here and in the main thread during the outage). Again, they are:
1) Why did it happen? What caused this extensive outage? (I know you say it was a corrupted server... but as an IT professional, that's not nearly a good enough answer. It's the ISP version of a computer repair person saying... "well, I guess your computer was hit by a virus" (when he really has no clue what happened). We'd kind of like some details... be as technical as you can. There are plenty of IT folks here on the boards that can help explain it to the rest of the people)
2) What measures are you implementing to ensure that it never happens again?
3) What compensation will you be providing us for loss of sales? THIS ONE IS IMPORTANT AND NEEDS TO BE DISCUSSED!!!!
Thank you for all that you do, and I look forward to the answers shortly.
posted on February 23, 2006 12:10:46 PM new
I would think that these questions would be better answered on an individual basis. I am surprised that you would seek compensation in an open forum. It is apparent that Vendio will not address these issues here. You could always write a real letter stating your concerns, and the factual reasons for your request for compensation.
posted on February 23, 2006 12:15:59 PM new
kozersky,
I am asking these questions because we were told in another thread (by Diane) that these issues would best be discussed after the outage. The outage is over... and we're looking to discuss it! They're very reasonable questions... and I'm sure I'm not the only person looking for answers.
Regardless... if they don't want to answer the questions in a public forum... they could at least state that, rather than just ignoring my questions.
posted on February 24, 2006 10:42:34 AM new
Will somebody from Vendio please answer my questions? (or at least let me know that your're working on formulating a response)
posted on February 24, 2006 02:04:54 PM new
Hi eauctionmgnt,
Didn't mean to have you feel ignored. Always a lot going on here, and it's often difficult to get to each individual question right away.
Here are the best answers we can give here in open forum. If you have additional questions, we can answer them offline individually.
1) Why did it happen? What caused this extensive outage?
A) The outage was caused by a hardware failure in the main ihost database. When the hardware was recovered and restarted (within 30 minutes), some of the data tables were found to be corrupted. To avoid more troublesome data loss or corruption, we decided to restore all files from backup, which are done very frequently (hence no lost data). I have to take some exception to "extensive," since this outage, including complete data restore, was approximately 2 hours. We could have been back in under 30 minutes, but that would have been riskier, and we don't like to take risks with your data. We have some process improvements that can bring the recovery time down further as well to under an hour in many cases.
2) What measures are you implementing to ensure that it never happens again?
A) Since you are an IT professional, I know that if I told you we'd never have an outage again ever, you'd laugh and never trust another word I said. That said, we have made some changes that will reduce the likelihood of another event and some process changes that will speed up recovery in case something does happen. Recognize that we have gone well over a year without a service disruption. You may notice we also don't even have scheduled weekly outages (and rarely scheduled outages of any kind). So 2 hours in 18 months is a pretty good uptime. I'm not saying we take any disruption lightly, but we have a strong reliability record that should be taken into account when assessing our infrastructure and desire to do our best to ensure these events don't occur.
3) What compensation will you be providing us for loss of sales? THIS ONE IS IMPORTANT AND NEEDS TO BE DISCUSSED!!!!
I have reviewed several accounts, and in almost every case, it has been incredibly difficult to discover any "loss of sales." I'm not saying it doesn't happen, and we will review the rare cases that there is some obvious impact.
In this case, fortunately images were up many hours prior to the normal closing time for listings (like yours in the 6:00-7:00pst time range). The outage was relatively brief considering the listing duration as well, of course, and we know historically that most bidding occurs shortly after listing or in the last 30 minutes (generally).
In your particular example, I compared your marketplace listings closing on the 19th with those on the 12th and 5th (comparable time periods). You actually had the most listings close successfully on the 19th (the day in question). You also had more listings that day (the 5th is a better comparison in quantity), but you actually sold a higher percentage of closed listings on the 19th than the 5th as well. All your items are nearly the same price.
I'm not saying we accept less than perfection nor that there aren't instances where we truly need to and will compensate someone, but often the perception of harm is much higher than any actual harm. It's a normal reaction to any disruption. If you carefully review listings yourself and see obvious issues you can demonstrate to us, you can submit a refund request to billing through the Email form on the Help page, and we will review them on an individual basis. We will respond to these requests as soon as we are able.
If you'd like to discuss further, let me know, and I'll send an e-mail to start the thread. Thanks.
posted on February 25, 2006 07:12:48 AM new
meffle,
Thank you very much for finally responding and starting an open dialog. I guess "extensive" is in the eye-of-the-beholder. When you're waiting for bids on your listings, and the images are down for 2.5 hours... you feel like minutes are hours. And, I do agree... I would rather have waited, then risk the chance of losing all my images for good.
I'm glad to hear that you are implementing some measures to help speed up and reduce the possibility of this happening again. I'm just curious, can you share them with us... or are they trade secrets? Also out of curiosity... have your engineers ever looked into Virtual Server technology? I just came back from a good conference that discussed this in depth, and it was designed specifically to deal with hardware maintainence and failures like this... where you could quickly move a server image from one server to another with no data loss (or even breaks in communication with users). This gives the opportunity to fix hardware problems (which we all know will happen sometime) without downtime. I'll be glad to dig through my notes and find more info on this if your engineers are interested.
As for the compensation, end of things... I have to say I disagree with the theory that there wasn't any loss of sales. I can't speak for others... but I can certainly speak for my own sell through rate. I would say it was down about 20-30% on my average sell-through rate. The data sets that you picked from my account were a very small sample. The weekend of the 5th had an unusually low sell-through rate. (That was superbowl sunday... so I don't think we should be surprised by that!) If you leave out that statistical anomaly and expand the data set (and compare my sales on the 19th with the 12th, 1/29, 1/22 and 1/15) I think you'll start to see a significant statistical difference. So yes... I do think that some sort of compensation is in order for my case. I know it's not easy to determine exactly how much of my loss your outage was responsible for... but considering I have been with your company since it started, and hope to stay with you for many more years, I don't think that a months credit on image hosting is too much to ask for.
I look forward to continuing this conversation with Vendio either here on this public forum... or, if you prefer, you are certainly welcome to email me with your response. Either way, I do hope to hear from you shortly. And again... I really do appreciate everything that your company does.
posted on February 28, 2006 03:47:30 PM new
eauctionmgmt,
My apologies for the delay. I'll send an e-mail to follow-up. I did forget about the Superbowl, but I generally compared results to the week of the 12th (non-superbowl), but your listing volume that week was lower than the 19th, so I went back to a period prior. More in a follow up e-mail.
.
Caroline 