As of today, SparkFun has switched to a new server - named Spino (after the largest known carnivorous dinosaur). Spino is a Dell R900 with 32 gigs of RAM and 4 Quad Core Xeon processors. The new server has allowed us to make a number of important internal changes to our systems, as well as one big upgrade for our customers - improved search.
For awhile now, the SparkFun website search has been...not great. But the new server has allowed IT to upgrade the search to be much more efficient. For example, searching "LPC21" now rings up with most popular LPC boards. A search for "Arduino" will bring up the most popular Arduino items. Before, searching for a SKU (or part of a SKU) didn't really work. Now, if you search for, say, "9573" it will bring up the part with the corresponding SKU (in this case, the Multimeter Kit). We think it's a huge improvement and we hope it will make browsing the SparkFun site easier.
Check out those pictures above. They're so real you can almost touch them! That's because they are actually 3D renderings done by SparkFun customer Jeff Miller. While the 3D drawings are very cool, the project itself is really what's more important. It is a programmer for iButton temperature/humidity data loggers centered around the Arduino Pro Mini - all built into a very nice enclosure. Read all about Jeff's project and his journey with Arduino here.
That's a ton of iron. I didn't realize ordering from a catalog involved semi-infinite computing resources.
I might need to upgrade the Gateway P-200 (Pentium-Pro 200MHz, 64 MB of RAM) web server running Apache, MySql, PHP and Subversion -- and also controlling my lights and irrigation system...
It's kind of fun to have such resources available, but did it actually solve anything that couldn't have been done in a more conservative, resource friendly way?
Get back to us when you start getting 250K pageviews/hr and hundreds of queries a minute to your sprinkler database.
Granted, it's none of my business. But if you boast about the size of your... hardware... in public, you're pretty-much begging for criticism. Mind you, I'm guilty of my own extravagances (like driving the automotive equivalent of that server, but at least it's 10 years old).
Oh, by the way, I'm back to you. Combined, and between the hours of 10am and 2pm Pacific Time, the websites I manage development for will FAR exceed that number of page views. And, hey, look there... my name's also on the WS-i Basic Profile specification (I'm Muir). Imagine that, I might actually know what I'm talking about...
I'm sorry you interpreted this news post as boastful. We're proud of the cluster we've built and wanted to share the details with everyone the same way we'd put any cool project on display.
Optimized code, more widespread memcached and better indexing on the database isn't something that can be turned into eye candy the way new hardware can. When Spino arrived we were excited to have a beefy new toy and we naturally assumed those following the SparkFun news thread might share that excitement. For fellow administrators it must be genuinely interesting what rigs other guys are building and for everyone else it must be just cool to look at a big shiny server.
Unfortunately the most vocal contingent out there appears to be largely critical and negative. I doubt we'll be as candid with future hardware upgrades.
I don't see your post as boastful, and I hope you continue to share these kinds of developments at SparkFun. I don't see the comments here as that critical and negative, and even if they are, so what? If not for your last sentence, I wouldn't have posted, and I suspect there are many more like me. The post was definitely interesting, inspiring, and more of a cool project than some LEDs in a CD case.
- Jan
I agree with jan. I love seeing what new "toys" SparkFun has gotten their hands on. I hope that you keep on posting about your new toys so that we know what to blame when something goes wrong :D
thanks for posting.
http://browsermob.com/instant-website-test/ce551b45-cb80-4c3d-87e9-1280297733b7/32136
That 500ms time to first byte, along with the cumulative latency of the multitude of requests, is killing you. Try creating a composite of the all the images on the page and use CSS to "slice" it into the various positions for each product. That alone will put your load time below 1 second, probably. Whatever caching you have in place might need to be tuned if it's taking 500 ms to send the first byte of what should be, essentially, static content.
Free advice, worth every penny.
code/db optimization ftw
Code and database optimization definitely. Also session, filesystem, cache, load balance, and hardware optimization, including better hardware where such improvements will make a difference.
Web administration is a rich tapestry.
Cool server. I have a few servers and wireless APs at home named after Star Wars planets...
Server 1: Tatooine
Server 1 iLO: Dagobah
Server 2: Endor
Router: Naboo
WAP 1: Coruscant
WAP 2: Alderaan
Switch: Hoth
Haha, eat your heart out Star Wars fans.
This is a very sweet upgrade! I'm a tech junkie and love to hear about server upgrades like this.
Is this server replacing the existing web cluster or expanding it? Just curious. Its good news either way.
Does anyone know how fast it can change from one frequency to another? I can't find this in the datasheet - I must be blind or maybe it only depends on how fast you send it operations.
Hm, interesting. A $3-5k dual nehalem xeon whitebox wouldn't be able to compare to this beast? She sure is pretty though.
So you wont be uing an arduino web server for the next free day then? :D
Don't tempt me.
RSS for this page is still horribly broken. All I get is the title of the post and, worse yet, the title isn't linked to the post. This is really frustrating because it makes my RSS reader useless for keeping track of news posts...
I submitted this twice now using the "feedback" portion of the page, but I've never gotten a response. For all I know, that might not be working either...
Two thoughts:
1. Could just be the intersection of your reader and glitches in the library we're using for RSS. I'd be surprised if it were quite valid RSS, although it seems to work for most people.
2. Maybe we're setting a MIME type wrong. I'll look into that.
A couple of days ago your RSS feed had a item called "Metallic Fish" in it, which looks like a post published at least 3 years ago (I forgot whether it was 2005 or 2007). There is something seriously wrong with your feed. It has been putting in partial items (items without dates, or bodies) since maybe a week or two ago.
Yep, and there it is again. I'll investigate.
FWIW, I'm using google reader. You can kinda see what's happening by going to the RSS URI in a browser.
This used to work flawlessly up until a few weeks ago. I've also verified that the URI is the same in google reader as it is on your site and they are identical from what I can see.
Thanks for looking into it.
Perhaps its your reader. RSS seems to work fine for me and the title of the post links to the correct article.
Why not Apache Solr?
Nice server, but I probably would have gone a different route. For the same price that you guys probably paid for that server, you could have gotten an Intel Modular with 6 dual quad core blades with 32gb each, and 14 10k sas drives for it's internal SAN. With a couple load balancers set up it would have made it through free day without a sweat :)
Anyhow, kudos on the new box, and have fun with your search engine.
Ill agree the search engine needs tweaking. I search for "connector" and instead of just the Ipod or smd connectors, I get every board for anything you guys have in stock that has a "connector" on it. Maybe you could add search options such as to search product titles only or the description too?
Aha, this is why the site was giving errors much of yesterday :P
Luckily I managed to place my order without problem toward early evening UK time.
I have been getting that "Something broke" page intermittently since early this morning, and again just now. =(
Something broke.
...not to worry. We've logged this error and will be looking into it.
getting this an awful lot tonight....
An astute observation. Should be fixed now.
Maybe now you can support next years free day! ;) wink.
First problem I see with your search is, if you search for MIDI, it gives you the top 2 results for Humidity. I would probably get rid of to part-of-word searches.
If search is so important to you, why not invest in a Google Search Appliance? Oh, that's right, because they cost $3000... But you could get one used for $1000. And it uses the Google PageRank? algorithm, so you know it'll be good.
http://www.google.com/enterprise/search/gsa.html
http://www.google.com/products?hl=en&safe=active&client=safari&rls=en&q=google%20search%20appliance&oq=&um=1&ie=UTF-8&sa=N&tab=wf
We vetted Google's various options for search but ultimately decided to go with Sphinx.
Why? Well, Sphinx was free, open-source, and very DIY. That's how we do pretty much everything here.
But there were a few ways where the long-term potential of Sphinx outmatched Google. Sphinx builds indexes directly on top of our database so we can fine tune how weighting and matching is performed to tailor searches to how we store data. With enough configuration we can perform powerful searches on anything in our database, even things that aren't individual pages such as customer comments (something Google can only index by the entire page on which it appears).
This is just the first iteration of the new search. Haven't yet figured out how to tell it to match both partial and full words AND give more weight to a full word match but I'm sure there's a way and we'll find it.
Wrong link:
http://www.google.com/enterprise/search/mini.html
(you should allow posts to edit, or at least delete)
Part-of-word can be kind of nice sometimes, but I would definitely make whole word matches higher on the list.