This is your brain on Dumpster Dive
A brief look at what happens when we offer Dumpster Dive boxes and why this slowed SparkFun.com to a crawl.
Don't forget – today is the last day of the New Year Clearance Sale! All discounts end at midnight tonight!
You, our customers, are fantastic. And of course, everyone loves a deal. When a passionate customer base encounters a deal, well, stuff breaks.
Let me introduce myself again, Double M, Director or Software Development and IT (SWIT internally). I’m responsible for ensuring our website stays up. When you, as a customer, can’t get to our website, I hear about it (normally from a few different places). It can feel like I’m in Office Space and I filled out the wrong TPS report.
Everyone was very kind, but I was informed that our website did so many concurrent orders that a bunch of our customers were hit in backorder or were given an ‘out of stock’ message as they tried to check out.
Our actual stock sold out in two minutes. That’s you guys. Fantastic. We sold product for two more minutes. That’s on us. We sold more in the second two minutes than in the first two minutes. Again, passionate people are fantastic. Everyone with a backorder will get their stuff, it’s just going to take us a bit of time.
For everyone who didn’t get an order, who got bounced at checkout or the page loaded too slow, our apologies, but we sold a lot more than we had to begin with. People were always going to get left out of this sale, there is no way to get these in the hands of everyone who wants them.
While we are an ecommerce website, most of our traffic is predictable. We rarely have to throttle; we rarely need to add resources into our production stack; we almost never block IPs or sets of IPs. In terms of raw sales volume and customer orders, Cyber Monday is a big day for us – we have a good sale and we have a lot of traffic. That is the benchmark in my mind. 100% Cyber Monday uptimes. We achieved that (not really, but we hit over three 9’s of uptime on that day (99.9% - just a few seconds of hiccup/sluggishness, not even downtime)), and had very few, if any, connectivity issues across our customers.
The back-of-the-envelope traffic graph shows that we had a mild spike over an hour, but most of that hourly traffic spike actually happen in a few-minute span.
Now my benchmark has to be what occurred on Dumpster Dive day. I don’t want to take the blame for all of the latency, but some of it is ours. We saw an orders of magnitude increase in traffic as noon local came about at the start of the sale. This stressed the systems but they responded very quickly – just not quickly enough. As we pointed out, the whole event was over in minutes.
We had a lot of processes and steps in place that we learned from last year, but the stress on the system far outpaced what we expected using last year as a benchmark. So even though we were ready to handle some increase over last year’s load, we had different problems all together.
Now, we have tools in the vaporware stage of development that may help us the next time we do this. Of course, now I probably need to expect another uptick in traffic again as one of our test cases. It will include tools we’ll talk about before we launch, and checks we won’t talk about, which should help level the playing field (if I told you, you’d know how to work around it).
I’m glad so many of you love the Dumpster Dive. We’ll try and make it run smoothly, but at its very core it is still a limited release flash sale. We’ll try our best to make it fair, but it won’t be a situation where everyone gets one. It can never be that.
Arghhh. I keep sending my comments to the "What's on your mind" !!! I always scroll down too far before commenting.
Aaaaaanywhooo.
"Everyone with a backorder will get their stuff, it’s just going to take us a bit of time"
WOW, what can I say ?? Fantastic.
I don't think I have ever come across a company that would honor the orders in this situation. In the past I have had orders cancelled and money refunded days after placing an order and getting confirmation, but not here at SF.
SF, you ROCK!!!
Thank you for the Hiccup's. I tried last year and lost but this year I got one! Wasn't sure with all the lag but after reading this I felt I should thank you Double M especially after knowing you have problems with your tps reports lol. I love being a a member of the sparkfun family.
We are happy to have you in the family as well. I hope you enjoy your Dumpster Dive box. Hope you come next year.
Very cool.
Out if curiosity what does your web stack consist of, and how optimized is it for scaling?
I was wondering what the backorder status meant on the half-pound box in my order. I'm happy that SF is going to the trouble to honor the orders instead of merely refunding.
I hope it helps that I wound up buying another $80 in stuff a couple days later. Oh man, what am I doing? This can't be good for my budget...
Enjoy your quiet day on Friday.
How do you check out with a backorder? I had to remove my backorder before I could check out with the rest of my order.
I think the concert ticket model will work well as it will spread some of the load out over time. An alternative would be a 1-click model. For those with cached payment info they click one button and the order is placed. It works particularly well in this "Limit 1 per customer" scenario. This option could be reserved for specific products. The benefit to SF would be a reduced load because the number of pages that have to be generated to complete the order would be reduced.
This comment is from someone who tried, got the product in his cart, saw very slow page loads, saw pages without CSS being loaded, and ultimately lost out. I should have considered my dumpster dive luck before I dropped cold cash on PowerBall tickets.
Here's a suggestion - just for these special events fix the shopping cart system so that once something is in the cart, it's not available for anyone else to grab. Get rid of the real world equivalent of people taking something off a shelf and then having someone else steal it out of the cart (or, alternatively, the equivalent of someone fraudulently advertising something for sale that doesn't exist).
I've heard all sorts of justifications that someone could lock up all the stock and you'd never be able to sell anything.So put a time limit on it, maybe 20 minutes, and only enable it for special events like DD. The stock will still sell out in 2 minutes, people will still complain about not getting the "good" stuff (there's a reason it's called a dumpster dive), but at least no one will be getting screwed by having their cartload of stuff disappear at checkout.
So this is basically the "Concert Ticket system" We are looking into this solution, one of the tools in vaporware right now.
It would be more like 5 minutes than 20, but it would all us to have customers store stock in their cart. Should help with payment issues and the like. We have a few implementation hurdles to overcome still, but look for it in the future.
On the dive - I missed out due to ticking "credit card" instead of "stored card". So instead I made a clearance sale order for similar effect ... and ended up with a cart at close to 72% off, and all stuff that I know I will use. A great fallback in my book, so no skin off my nose missing out.
Now get to decide what gets the CUHead DNLA audio capability (sale made it the same as a "small dive": a fab deal). Currently the AM/FM/LP console with the tubes is in the lead (it has some RCA inputs in back), because more tubes and it's winter, so curl up by the stereo for heat, eh?
I'm glad you were able to make the most out of the clearance sale.
As was mentioned the pages loaded sloooow.
If you would like an estimate as to when your specific order will ship please email cservice@sparkfun.com and they should be able to look up your order number. As for the "someday" well we have to find more stuff... We actually held back almost 50% of the stock because we knew we were going to end up with backorders and we hoped that would fill them. That is why some backorders can be be filled right away. With our inventory day coming up this week we might not have have time to find and build more boxes this week. It is definitely a priority, but it might take a week or so.
Thanks for answering my concerns. Have a nice day.
I get the SF site latency and that all makes sense. But is there any particular reason paypal failed to respond? In my case, that's what happened. I had it in the cart, clicked the paypal, got an error, returned to SF and it was out of stock. Any ideas there?
geesh. just got my boxes. i paid 30$ for 1 pound of aluminum standoffs and screws, two usb cables, a book about Arduino, a million 1/4 watt resistors, 2 six axis breakout boards, and a sound cape circuit board!!! last year was so much better...
anyone complaining about not getting two boxes can send me 10$ and ill ship you mine...
Done. What's your address?