Snappy Cluster
The cluster has arrived to our new data center! 5 Dell PowerEdges with 10 megabit burstable to 100 megabit. We're trimming the fat off our site and making things much faster.
Happy Holidays! We are closed Thursday December 24th and Friday the 25th for Christmas.
In just enough time for the quiet X-mas holiday, we've got our new servers up and running.
In just enough time for the quiet X-mas holiday, we've got our new servers up and running.
Ben installs our 5 Dell PowerEdge R410s
We're going with a dinosaur theme: Tyrannosaurus, Magyarosaurus, Stegosaurus, Ankylosaurus, and the coolest sounding Psittacosaurus.
Ben and Brennen are happy campers. Chris was there as well shooting photos on his Android.
What does this mean?
First, we've got a larger pipe to the outside world. This means we can handle 10 megabit standard with the ability to spike up to 100 megabit or more.
Second, this means that we have successfully taken the leap from a dual server system (one database server, one web server) to a 'cluster of nodes' system:
Third, distributed computing means that load handling is not linear. We did not go from 2 servers to 5 servers - we went from "2" to "20" effectively. We're not sure we'll be able to handle the full load of Free Day, but we're arming up as best we can.
Admittedly the SparkFun site was molasses since the week of Thanksgiving. We hope that you now find the site snappy and quick. Please chime in below - we'd love to hear what you think of the new site.
We've decided to team up with RockyNet. Their data center was secure, clean, well maintained, and provides a physically redundant, dual gig-e optical connection to the outside world for about $1400 a month. This includes a tap onto the massive UPS+generator system and an alternating hot/cold aisle climate control system. It was a balmy 60 degrees. Ahh, room to grow.
We're going with a dinosaur theme: Tyrannosaurus, Magyarosaurus, Stegosaurus, Ankylosaurus, and the coolest sounding Psittacosaurus.
First, we've got a larger pipe to the outside world. This means we can handle 10 megabit standard with the ability to spike up to 100 megabit or more.
Second, this means that we have successfully taken the leap from a dual server system (one database server, one web server) to a 'cluster of nodes' system:
- webserver
- load balancer
- static content server
- dual load-balanced database servers
Third, distributed computing means that load handling is not linear. We did not go from 2 servers to 5 servers - we went from "2" to "20" effectively. We're not sure we'll be able to handle the full load of Free Day, but we're arming up as best we can.
Admittedly the SparkFun site was molasses since the week of Thanksgiving. We hope that you now find the site snappy and quick. Please chime in below - we'd love to hear what you think of the new site.
So those Sparc servers? lol
If the SparkFun website was like molasses before, now its like butter :).. Possibly even melted gooy running chocolate.. mmm, chocolate. Loaded up nearly instantly and I was greeted with server porn, mmm... Looking great! Love the orange cables, too.
-robodude666
Wow, so your servers are dinosaurs already? shame.
Ahh the Dell Poweredge series. They are beautiful machines. My personal favorites are the 2950 series. Team a few of those up with a Dell PS6K storage appliance and you have yourself an unstoppable cluster. Its nice to see that you guys are able to keep up with the demand. Your site is fast and everything is working beautifully.
The site is WAY faster than it used to be. Hopefully, Free Day doesn't cause the servers to spaz like an epileptic on crack.
Just my 2 cents here, why not go inside a cloud hosting environment like amazon or the newest rack space cloud?
This way you could add resources as you grow without tempering with servers, resources and so on
Anyway, very nice setup :) Good luck!
You did a lot to beef things up, but I think you can go farther w/o breaking the bank. The good news is that infrastructure optimization usually requires little in the way of code change.
You don't mention memory/OS/DB; based on what I know, I'd spec this:
OS - 64-bit Linux (ext4, noatime) or OpenSolaris (ZFS), with a decent swap.
Memory - Double it.
Add memcached servers - store session/shopping cart info, product catalogs, common or long db queries
DB - Add indexes on columns in your where clauses. Do explain plans and tuning analysis to optimize long-running or commonly used queries. (Google for more).
PHP optimizations - XCache and memcached (XCache is more important than ZendOptimizer)
PHP API - Turn on (redundant) memcached servers and "static" page caching
Test, do load and db analysis...
With another 40-160 hours of work, you can be ready for almost anything, without major hardware expenditures (memcached servers can be cheap)....
You definitely need to add a tier 1 disk array with about 12 disks in raid 10, at least for the SQL DB. I'm betting a majority of problems that you saw today are from disk IO bottle neck. Also 1U servers are almost pointless,unless you cluster, when you can use virtualization instead.
-$0.02
i see about 10 more of those needed for the '2nd annual free day' (hint hint) lol. site is running smooth and snappy but with so many ppl on ,i was on for the nearly 2 hours and never made it past the login page :-( without timing out and having to refresh 87 times) im curious to see all of your statistics from this morning and see where the weak point was, but assuming a stick of ram didnt catch fire or a network cable melt, i's say your new cluster did amazing considering!
Five servers and it took you almost 2 hours to handle what was probably 1000 or so transactions? So you needed five physical servers and 10Mb of bandwidth and still you could only process one order every 6 seconds at best?
You guys should fess up. You throttled the web server to drag the promo period on for much longer.
Probably should have beefed them up a bit more more Free Day. :(
The site is too slow...Look at Alexa.com stats for your site...96% of sites in USA are faster than yours. Pls..add more web servers, cache more contents in memory and partition your databases....The biggest performance killer could be your disk systems...invest in a SAN - they are getting cheaper these days.
Perhaps I'm being snobby, but an SMC switch in your datacenter? Sad Panda. Also, are those teamed NICs or are you actually running the data and mgmt NICs to the same switch? I would hope not......ideally you'd want 3 switches, two data's and one mgmt.
Are these just web servers or are a couple of them SQL servers? Seems fairly small for both if that's the case. To put things into perspective, one of our SQL clusters (serves about 4k employees with various medical software/EMR DBs) is a full rack of Dell's connected to another full rack of iSCSI storage. I can almost smell the smoke of tomorrow's hellfire these servers are going to be producing. lol
Joking aside, you guys have grown amazingly and I wish you the best of luck tomorrow and all throughout this next year! Keep up the awesome work guys!
Try sharing the webserver role across more than one server if things slow down?
"SparkFun site was molasses since the week of Thanksgiving."
And it still is right now a day before Free Day.
That's gotta be fun to work with and nice to have.
Awesome looking stuff!
While you guys are modifying the website, perhaps you can add a wish list feature.
Good job on the website.
Can't wait till free day!
I swear there is some kind of conspiracy going to have someone ask for this at least three times a week.
In all seriousness, we'll put one together one of these days. Promise.
The new servers look awesome and site response does seem better then usual. I was just sending a gift certificate to my brother and typed up a message with several paragraphs and when I previewed it the text went back to block format. Using the back button on your page to insert some HTML erased the message. (Thankfully, the browser's back button brought it back!)
Maybe some built in HTML support would be nice, or in the spirit of Craigslist keep forcing us to do the work and use our brains a little more!
Congrats, and happy holidays...
Friggin awesome work guys! You're my hero's.
Mmm, I could use a rack of THOSE for Christmas...
Big up for MAGYARosaurus :)
Vic from Hungary (as in our native language: MAGYARorsz?g) ;)
Great work guys.
Wonderful! Everything loads quick, with good reflex to move from product to product.(even while displaying all items for categories!)
Ok, I know you're hardware guys, but I'm curious why you went with your own machines. Why not go with something like http://aws.amazon.com ? Then, you can trivially spin up extra servers for events like Free Day as load groes and pay for only the 24 hours you need them, instead of having extra capacity you don't need most of the time.
(Note: I have no affiliation with Amazon other than happy customer.)
I was going to ask the same question. I'm interested in Amazon's services. The site is loading fast, I can't wait to see how is does on free day.
My company has used EC2 a ton and am pretty unimpressed all around - at one point we've had as many as 30 instances running with about 20 of them being the largest they offer. While spinning up new instances on demand is very easy once you have the right scripts, the actual performance of each instance is quite poor. We've seen something on the order of 50% improvement under load from similar real-world hardware vs. the EC2 virtual image. It's not at all consistent either - the performance fluctuates wildly per machine and time frame.
We made the terrible mistake of assuming performance on EC2 was reasonably close to what it would be in production and spent a small fortune optimizing the system when it was totally unnecessary.
Spinning up servers on demand for load is cool, but running everything on EC2 alone just isn't quite there yet. Ultimately it's more expensive too if you use a fair bit of bandwidth. As it stands, we are only going to use EC2 for times when we need a machine RIGHT now and we only need it for a few weeks to a month or two.
"Trivial" is an interesting word. As with terms like "simple", "easy", and "cheap", it is not exactly resolution independent.
In all seriousness, there's almost certainly a place for something like AWS as we scale up, but we weren't / aren't there just yet. Meanwhile, it's a nice baseline to own and control the hardware we need right now. Especially with certain deadlines looming.
It does load a lot faster, and the feature to 'Click for Larger Image' is working again for me (latest FireFox on Windows 7 RC... It worked well until Thanksgiving, more or less). However I can't still select other images unless I switch to IE8.
Hmm... There's something wrong, a lot of pictures aren't loading for me.
So much faster! I have a pretty slow connection, and sometimes I had to reload pages to get them to even come up. Now it's instant, and I don't have to spend three minutes getting to tutorial 3 so I can figure out how to wire my atmega168... pin 9 and 10 go to the crystal! Nice work guys!
The improvement is easily noticeable and greatly appreciated. As of late the site was getting pretty slow.
i need help getting my jaw of the floor!
THIS IS AWESOME
Hmmm... I was wondering why the website loaded so quickly... Anyway, with all the electronic things you have, I'm a little suprised that you didn't build your own servers! :) Have a good Christmas!
I used chronological satellite / spacecraft missions for laptop computer names.
Sputnik was an old laptop, Viking is Dell Inspiron.
Writing this on Magellan (aluminum unibody MacBook)
It's interesting how other people name their computers.
I use a moon theme. My laptop is 'mare-nubium', server is 'magnificent-desolation', and my desktop is 'tranquility-base'.
=)
I have all my systems named after William Gibson ideas.. Wintermute, Neuromancer, ChibaCity, Turing, Straylight, TheFinn... The list goes on... and on and on... :)
That's a pretty sweet theme, although the set of William Gibson ideas is not exactly the first thing that comes to mind for Turing. ;)
We debated themes for a while... I think Brennen came up with the dinos and I couldn't be happier with it. It's a nice rich field to pull from!
When I first started at SFE I had to personally manage all the computers which were very disjointly named. I too enjoy all things space-related so I started naming things after celestial objects. I'm currently typing this on Hyakutake.
Then there's software releases. This fall we benchmarked the SparkFun codebase (known around these parts as SPORK) and started up a fresh naming scheme based on the last names of musicians that don't suck. This site is currently running Danzig and Everett is due out sometime in January.
Naming is fun!
I think that must've been a Ben & Casey idea. We spent some quality time with the Wikipedia list of dinosaurs somewhere in there, too.
Very snappy so far, It does seem faster not that it was all that slow to begin with
Psittacosaurus = taco-saurus in my brain from now on. :)
Fantastic upgrade job :) makes shopping a much more pleasurable experience.
Hmmm, I may have to resubmit my end of contest time.
Clean wiring rocks.
Why not Dell's DRAC instead of the KVM connections?
The datacenter is just down the road. Maybe we'll move to a DRAC setup in the future but up front it just wasn't all that necessary, considering the cost.
Nice X-Mas spirit, and I see this years toys are really neat :)
Happy Holidays guys!