I love my job. After 11 years I still learn something new everyday. Most days it’s a fun experience, some days it’s painful. On Monday we learned we had shipped as many as 1,934 MicroView units without a bootloader. This renders the unit effectively broken.
To the folks that are receiving broken MicroViews over the next few days and weeks: We are really sorry to have messed up your first impressions of SparkFun, Geek Ammo and the awesome MicroView. We are going to make it right and will be shipping a replacement MicroView for every defective unit that was shipped. For those of you who backed the MicroView Kickstarter you will be contacted by September 12th to let you know if you were or were not shipped a defective unit. We are still trying to establish how many units are affected. The worst case scenario is 1,934 units need to be replaced outright. If you are part of the defective batch you will receive two units: one that has a broken bootloader (now and in the coming days) and one that works (by the beginning of November). If you’re willing to try, this is the perfect opportunity to learn some new skills: soldering and bootloader programming. Success means you’ll score a second working MicroView, free of charge.
To the folks that received MicroViews prior to July 18th and all backers of the Learning Kit tier, enjoy them. You should have a fully functional unit.
To the folks that just happen to be reading the SparkFun post today, let me tell you about a $58,000 lesson in mid-volume manufacturing.
How can I tell if I have a defective unit?
The folks at Codebender have been rock stars and helped us out. Checkout their simple Microview test page. You'll have to install their plugin.
If you'd rather use the Arduino IDE, follow all the steps to load a test sketch:
- Install the FTDI drivers if necessary
- Attach a MicroView to the programmer
- Select Arduino Uno from the Arduino IDE drop-down menu
- Open the Examples->Blink sketch (or any sketch)
- Upload
If you follow all these steps and get the error:
avrdude: stk500_recv(): programmer is not responding
or
avrdude: ser_recv(): programmer is not responding
or
avrdude: stk500_getsync() attempt 10 of 10: not in sync: resp=0x00
Then you know you’ve got a MicroView that is missing its bootloader.
Who do I contact to get a replacement?
You don’t need to contact us, we will contact you if we find you’re in the defective batch. If you’ve got questions, concerns, comments, please send them to microview@sparkfun.com. We want to hear from you, whatever it is, but please give us at least a few days to respond.
When can I expect my replacement unit?
We’ve ramped up to fix all the MicroView units we have in the building, build new MicroViews, and get you a replacement as quick as we can. We’re aiming for the end of October, early November. Again, we’re really sorry to have messed up. Please sit tight while we build more MicroViews.
Can I fix the bad unit once I get it?
Yes you can, but it’s not easy. In the next few weeks we will post a full tutorial to show you how to reprogram and recover a defective MicroView unit. In the mean time here’s a breakdown of what is required.
First you need a programmer capable of programming an ATmega. If you have an Arduino, you can use it as a programmer. If you want a cheap programmer in general the Pocket Programmer or the Tiny AVR Programmer are a great option.
Next, six connections are required for in-circuit serial programming (ISP). Three of the connections are located on external pins (easy to connect), and three of the connections are small vias on the internal PCB. This means the end user has to open the MicroView unit and attach (by soldering or holding) three wires to vias on the board, as well as attaching three wires to the exposed pins.
Next you need to run avrdude with this specific HEX file to burn the new firmware (that includes a bootloader!) onto the ATmega328. Once the firmware is loaded make sure you can upload sketches.
If everything is good, disconnect the ISP connections and carefully repack the MicroView into its housing and snap the lens back into place. Do a little dance, take a photo and tweet it @sparkfun because you are so awesome. We will high-five you back.
How did this happen?
Over the past few months we have been building approximately 8,000 units for the wildly successful MicroView Kickstarter. On July 18th, 2014, a new production firmware was created to better test the units. Unfortunately this new test firmware was defective and didn’t include the STK500 (aka Optiboot) bootloader. The test procedure correctly tested the MicroView’s functionality (display graphics, toggled GPIOs, etc) but did not test the upload functionality. There is a reason for this: enumerating a COM port and uploading a sketch is much slower than pushing production firmware over SPI. Since 2011 we’ve been streamlining production by combining HEX files. This means we combine the HEX of the bootloader and the HEX of the production test code into one HEX file that gets programmed onto the final unit. We build nearly 90,000 products a month. This approach using combined HEX files has worked swimmingly for years. But you can see the Achilles' heel - if the HEX gets incorrectly formed it can be difficult to detect. On July 18th we started programming units with defective firmware and didn’t know it until August 17th, when customers started contacting us about the problem.
What’s broken?
There are (worst case) 1,934 units that got programmed with test firmware but lack the STK500 (aka Optiboot) bootloader. The units are fully functional and should display the test sketch just fine. The problem is that there is no bootloader so you can’t upload a new sketch.
Am I in the bad batch?
We build batches of MicroView 128 pieces at a time so there’s actually a large number of batch identifiers. We’re still trying to pin down the exact batch numbers (there’s a lot of them) and how many units are still in the building. We’ve been doing a great job of building and shipping units ahead of schedule but we have a mixture of batches still in the building. Once we get everything figured out we will notify the backers that are affected. You can easily test your unit following the description above.
What did we learn?
1) No matter how much it costs, make it right with your users. 1,934 units * $30 for the replacement unit and shipping worldwide = $58,020. This sucks, and we screwed up, and we’re going to do everything we can to make it right. Sorry for messing up that moment of joy when you get your MicroView. We’ll get a replacement to you as quick as we can.
2) Don’t change production firmware mid-run. We’ve built hundreds of thousands of products, but in small batches. This is the first time that the stakes were so high. In the future, if we have to change the firmware we’ll send the new firmware through beta testers to make sure.
3) Test the bootloader. In the past we have not tested the bootloader because it required that we enumerate a COM port (which can take as long as 30 seconds) and send a test sketch (another 5-10 seconds). After the MicroView error was discovered we quickly agreed that we should be testing to see if a bootloader is present. This test can be done by resetting the ATmega, sending characters 0x30 0x20, and waiting for a 0x14 0x10 response. This test can be done with a second ATmega on the test jig itself (no computer required) and adds very little time after the programming step.
4) Moving from low volume to mid-volume production requires a very different approach. SparkFun has made this type of mistake before (faulty firmware on a device) but it was on a smaller scale and we were agile enough to fix the problem before it became too large. As we started producing very large production runs we did not realize quality control and testing would need very different thinking. This was a painful lesson to learn but these checks and balances are needed. If it didn’t happen on MicroView it would have happened on a larger production run someday in the future.
In 12 days SparkFun moves to its new building. I love my job: a new lesson every day.
For those who have bad units and want a head start on trying to install a bootloader here is a generic tutorial on installing Arduino bootloaders. We should still be coming out with a Microview specific tutorial showing you where each pin is, how to open the case etc.
Bootloader loaded and working. When it comes time, SF should ask if the recipient wants a replacement or not. Personally, I do not need a free unit; all is well. I used the end of plastic zip-ties (versus something metal) to pry the enclosure open as not to damage anything.
I've written up a walkthrough of how to fix your MicroView for MAKE. It takes you through opening the case, uploading the bootloader using an Arduino as an AVR programmer, uploading some test code from the Arduino IDE, and then closing the case back up afterwards.
That was quick and excellently documented. Thanks!
That definitely came in handy, wished you showed more of popping the bezel off, it worried me but it actually wasn't bad at all. I made a quick video of popping off the bezel, soldering jumpers, and then programming in AVR studio http://youtu.be/S161IQ9rBrM
Clear and informative! Thanks!
I noticed a typo: "Go ahead and put detach ..." extra "put".
Super useful, thanks!
Great stuff - thanks a bunch.
Props for telling us what's going on and this is the right way to deal with this situation. Thanks Sparkfun.
Hopefully the employee(s) that fubared the mid-changed firmware aren't being given such a hard time cause I know from experience of how such a simple mistake can easily happen, but luckily I don't deal with such a high volume of boards though. ;)
Yea, one person clicked a thing but really it's SparkFun's fault, not one person. This was an error that got past multiple departments and multiple people. A faulty bootloader run could have happened from anyone. Now we're putting things in place to make sure it doesn't happen again.
I totally get warm fuzzies when I read this.
I've made a few mistakes professionally and seen others do the same and I've never seen a CEO take so much of the responsibility on the company instead of a single employee. It's like you're all on a team or something!
My first week at SparkFun I somehow deleted an entire product page. I have absolutely no clue how that happened...
I went and talked to him yesterday because he's now in a special club. Some people call it the "$40,000 Club" because in order to be in it you need to be the person who triggered a situation that cost around that much, or more. Certainly shipping out 2,000 new Microviews will probably qualify.
I joined the club way back in 2008 when I was working on our credit card API implementation. We were way less serious about testing and code review back then, and I committed the credentials and path to the testing site instead of the live site. For two days all credit card orders just went through as long as the card passed a Luhn check, because they were all hitting Auth.net's sandbox, so no money was actually changing hands. We found out after two days when Finance was expecting the usual big (back then ~$70,000) deposit to show up in one of our accounts... and it didn't.
In that case and in this one it's one person who precipitated the issue, but the failure was systemic. It's a learning opportunity and fortunately the folks at SparkFun are cool and professional and fully aware of this.
See also. And also.
I dropped a "pod" of wafers while I was working at TSMC (Taiwan Semiconductor Manufacturing Company). All 8 wafers shattered. I lost several months of bonuses because of that...
I'm sure a lot of us here are members of that same club... I, myself, was promoted to a much larger $$$$ club. ;)
OUCH. I'm lucky I've never done that. I work on billing gateways a LOT, and I've managed not to commit that faux pas. I did once issue this command on a production server: TRUNCATE TABLE order_line_items;
Luckily we had backups from a few hours earlier, and copies of all the invoices for that day. I got to practice my data entry skills.
I thought I would share here how I re-flashed my microview. Here are some pictures of my re-flashing setup. Upgrade microview firmware pics. It's relatively easy to re-flash if you have access to a programmer.
After removing the lens lift us the display and locate the via's labeled 11,12,13. These need to be routed as follows:
TP:13 -> SCK -> ISP Pin 3
TP:12 -> MISO -> ISP Pin 1
TP:11 -> MOSI -> ISP Pin 4
Then on the external pins
5V -> VCC -> ISP Pin 2
Gnd -> Gnd -> ISP Pin 6
Rst -> Reset-> ISP Pin 5
For my avrispmkii the following avrdude command line worked:
avrdude -P usb -b 19200 -c avrispmkii -p m328p -v -e -U flash:w:MicroView_combined_8-19-14.hex -U lock:w:0x0F:m
With the firmware from MicroView Firmware.
Your mileage may vary but hopefully this helps someone.
Can't view it.
Most people should have access to a programmer potentially. I needed an AVR programmer. So I went to Radio Shack and bought a USB Arduino, a project box, some leds, and some connectors. In a few minutes I had assembled the arduino in the box with connectors labeled for programming. I loaded the firmware that comes with Arduino software for programming AVRs and I was set. Run avrdude like you indicate and you are ready to go. In the past I received a blank (no firmware) MEGA8U2 board from Sparkfun. That is how I got the firmware on there.
Worst case is you end up with an extra Ardunio laying around the shop. I just don't see how that is bad at all.
Oh, and the whole NOT trying to hang someone for a mistake attitude is another reason I am glad I buy my stuff from Sparkfun. Thanks for being a bit awesomer today Sparkfun!
Anyone in the greater Bozeman Montana area needing to flash a new bootloader? The Bozeman Makerspace has a programmer and is willing to assist ...
Giving support from Chile!
I've have the tiny AVR programer. When the full recovery tutorial becomes ready, I can help local users to install their bootloader. Maybe whe can make "local" task forces to help.
I'm new to the open electronics world, and have never ordered anything from SparkFun. This blog post makes it significantly more likely I will purchase something from you in the future. I'm guessing this incident, and most importantly your response to it, will more than make up the cost of the defective units.
It is obvious that you haven't lost touch with what it's like to be a client, and not just a customer, and how you would like to be treated in that type of relationship. I had a boss who used to insist we refer to the people and businesses who purchased from us as 'clients' rather than 'customers'. He said the difference is that 'customer' implies a transactional relationship - I pay you money, and you give me a good or service. At that point, the relationship is over. 'Client', on the other hand, implies a long, trusted relationship.
Regardless of the language you use, it is fantastic to see a company respond in such a respectful way.
http://www.hobbyking.com/hobbyking/store/__27195__atmel_atmega_socket_firmware_flashing_tool.html
That is a link to a firmware flashing tool, it is used to upload custom programming onto atmel chips. It is done by holding the socket on top of the chip and activating the uploader. I believe this would work to burn the new bootloader onto the chip, but I am not sure. I am also unsure if there would be easy access to the atmega328, or if soldering onto the 3 pads on the PCB would be the only way to get access to the pins.
Awesome. Sparkfun could even give people a choice of shipping them this cable or a new unit!
That is a truly useful cable! Thanks for the link!
Thanks, Nate. Just the simple fact that you opened and closed this post with "I love my job" is exactly why your company has been so successful. Don't ever change, young man!
This is an awesome lesson in the right way to handle a problem. My opinion of SF has gone up because of this. I'm actually hoping I got one of the defective ones -- to give me an excuse to open it up and try to re-flash it. Thanks for doing things right, guys -- and thank you, Nate, in particular, for calling this a SF problem, rather than throwing the "one guy who clicked a thing" under the bus. You're good people.
Only a Sparkfun customer would say, "I hope I got one of the defective ones!" :)
Thank you! Monday was really rough but comments like yours make today and tomorrow look a little better.
Update: Got mine today. Apparently Batch 53947 is one of the good ones, since it can be reprogrammed with no problems. I guess I'll just have to buy that second one, at some point. "Darn." 8-)
I was able to burn the boot loader to mine Too. Simply hooking the ICSP (MOSI, MISO, SCK ) directly to the ATmega TQFP pins on the back of the PCB, and then power, ground and reset pins on the Micro view header.
But for me all I did was go to my Arduino IDE, select the board => Arduino Uno. Then select burn boot loader. I used this programmer this programmer https://www.sparkfun.com/products/11801. All fixed.
Then disconnected it and hooked the MIcro view up through serial pins and an flashed the cube script.
Everything is working great.
Can't believe you are going to send me another one for Free! So cool 2 for 1.
I would add that another lesson is to always, always include SOME kind of ISP footprint somewhere, even if it requires a special pogo pin adapter to use it. And unless space is really at a premium (like it likely is here), it should be the standard 2x3x.1" footprint.
This wouldn't have been quite as bad a disaster if the recovery path didn't include picking out vias for tacking on programming wires.
I'm not this guy, but this gizmo is a great way to go, IMHO.
Like our pogo adapter? ;)
You're right - we try to maintain ISP footprint on anything with an Atmel on it. We really had to trade space for ease of reprogramming on this one.
Also, because some of the ISP lines are used to control the screen, it made more sense to put "free to use" IO on the headers. Although it will be time consuming no matter what we do, we have already found one solution that makes reprogramming a little easier for us. We were able to modify our current programming pogopin adapter to contact the ISP lines while still keeping the PCB inside the bottom piece of the enclosure. This enables us to reprogram without entirely disassembling the product. Every little thing helps!
Well, I'm guessing your pogo adapter (which I have and use nearly daily, I'll add) - which uses the standard .1" footprint - wouldn't have been usable here because of space constraints.
The one I linked is a 1/4 scale version and only two of the pads need to be through-hole.
Now, not using a standard footprint may not have helped if customers needed to do it, but I do wonder if ramping up production on a whole mess of remediation kits (consisting of the appropriate ISP pogo adapter) might have been less costly than ramping up production on a whole mess of replacement units.
Thank you once again for your continued openness. I recently received my Microview, played through the on-board tutorials, but have yet to upload any sketches. But whether or not I received a ~~faulty unit~~ unit on which I can learn more, I will continue to buy as many Sparkfun products as I can, in order to help you quickly recover the losses incurred. Yes, that's it, that's why I'll be buying so much stuff.
Tried to upload a sketch, and this is what I got:
I'm excited for the learning opportunity, but sorry that it happened this way. Thanks again for your openness. Nate, you are my Queso Grande Hero!
Wasn't aware of the Microview until you tweeted about this problem - now I think I will be buying some in the near future as it looks like it will solve a specific need at home - Serendipity at its best :)
This article, not by me btw, may be of some interest - the difference between testing and checking
Everyone makes mistakes: it's how you recover from them that matters.
I wish CEO's from bigger business like GMC etc read your post and watch your company closely. If they were the consumer's world would be a better place and they'd get the customer loyalty that they can't find and don't deserve.
My Microview (which shipped and arrived last week) is unaffected by this, but thanks for the heads up!
I received one of the faulty MicroViews and have managed to install the new firmware. I do not have an AVR and I am running a mac at home so I had to stray from the normal procedures a little bit. I created a guide to how I got it to work for me using a mac and an Uno. In the end it was fairly straight forward. http://metaphorforwhat.wordpress.com/2014/08/22/microview-bootloader/
Good work to SparFun and Greek Ammo on how you have handled the situation.
Very nice writeup, with explanations of what needs to be done and nicely leveraging the existing Arduino framework to burn the bootloader without having to directly deal with avrdude.
5.For ANY product run ensure there is a FULL manual check following reduced AQL levels. This should follow a user orientated eye view. ( that means for a 50,000 pcs order you FULLY function check 20-40 pcs manually)
In my past, we once produced 500,000 'lego' type talking space men who were supposed to say: 'XXXXX blast off' . Instead the sound chip suffix was wrong and a large % said: 'xxxxx F*** off' .The IC foundry mixed up the order with a novelty card order. ( you know this happens a lot, by the number of children's toys you see in the press doing the same). Automated testing and Chinese workers cannot spot this.
Yesterday I started out with two shiny new defective MicroViews, today I have two shiny new fully functional units, plus a lot more experience with uploading bootloaders. I had a USBASP programmer lying around which I purchased sometime in the past without real knowledge how to use it, and it turned out to be a great decision.
Before using it I tried using an Arduino as a programmer, but I couldn't figure out the correct avrdude parameters, so I decided to give the programmer a shot.
What I did was the following: I am on a Mac, so I installed the CrossPack-AVR package and downloaded the .hex file mentioned above.
I opened the case with a small knife, pushing the lens to one side. It comes out this way with no visible damage, but I had to be very careful. After this, I lifted the top of the screen a away from the PCB to reveal the additional 3 holes I need. After examinigin what Ben Kearns (Thank you! :) ) wrote here yesterday I used the following routing: Programmer Pin 1 (MISO) -> Pin 12 (inside) Programmer Pin 2 (SCK) -> Pin 13 (inside) Programmer Pin 3 (Reset) -> Pin 1 (outside) Programmer Pin 5 (MOSI) -> Pin 11 (inside) Programmer Pin 9 (Ground)-> Pin 8 (outside) Programmer Pin 10 (+5V) -> Pin 10 (outside)
After setting this up and connecting the programmer to a USB port I navigated to the directory where the .hex file was located at, and I ran this command: avrdude -P USB -b 19200 -c usbasp -p m328p -v -e -U flash:w:MicroView_combined_8-19-14.hex -U lock:w:0x0F:m
This did the trick, and now both of my MicroViews are functioning fine, running different sketches. I hope this helps someone out, as I was a complete novice when I started to do this.
Thanks to the pointer to CrossPack, that's a nifty package. Another possibility is to use the avrdude that comes with the Arduino software (which many MicroView users are likely to have installed). It's a little tricky to find, but lives here: /Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/bin/avrdude
Today's lead posting by Nate was a breath of fresh air and interesting reading during my lunch break. Human endeavor. Human failure. Human touch. Nicely Done!
By chance the Microview package was waiting for me this evening. I happen to have one of the 1,934 Worse Case family. Not a big deal, I contributed to the Kickstarter back in March, and have spend very little time thinking about it since, as life has a habit of filling my 'spare' hours.
On the bright side I was delighted by the assortment of extras in the Sparkfun box! The label is "DEV-12853" but it's nowhere to be found on Sparkfun.com Care to bring me up to speed? Do we have a packing list or BOM? Is there a tutorial for this kit?
The SKU on the box is basically our internal SKU. Because these were a Kickstarter item they don't have a page on our storefront. I believe the kits should have a piece of paper with a website, otherwise check out http://learn.microview.io/.
M-Short, thanks for the pointer. Looks like I have the Microview Learning Kit. Your pointer took me to the eleven tutorials for the kit. Now I'll try to carve out some time to run through them. Firstly, I'll try to replace the bootloader.
I received my MicroView yesterday, and confirmed that it has the bootloader. Is there a way I can provide the batch number to help your troubleshooting?
Depending on which kit you got, you might find a small white sticker on the side of the box with a barcode and a batch id underneath that. I doubt it'll change our procedure much at this stage, but I'd be curious to know that number anyway, if you happen to find it.
Batch 53973, bootloader working. One of the pins on the programmer folded up when it was placed in the breadboard for shipping, but it's already back in it's proper place.
I got two from batch # 54579, both good as gold.
The batch number from my box is 54108
I got a defective one (this is not a support request, just a statement of relevance). Looking forward to the microview reprogramming tutorial, as I'm a novice, but an ambitious one.
As a Denver resident, I'd be delighted to save you having to ship me the replacement when it's ready. I suppose that's logistically complex, but if there's a way to make it happen, let me know, and I'll be the first to sign up.
Congratulations on remaining my favorite company in the world. It's a small accolade, but heartfelt.
I'm all in for it, and I know a couple other local SparkFans who would likely love another excuse to stop by your space. Perhaps Geek Ammo would cheerfully send out a link to an opt-in form for pick-up? We might be able to save you a full dozen dollars in total!
Done for yours and Tekee's. If anyone else would like theirs switched to local pickup while we're waiting for replacement stock, hit me up with a reply to this comment or mail to [this alias at sparkfun dot com].
Of course you know I am on the local pickup list, as I've already done it once with your help. Thanks Brennen!
I commend you for doing the right thing, without a public outcry first.
I hate to tell you I told you so, though, see my comment to this post, https://www.sparkfun.com/news/1352 that validating test fixtures is an important part of production. Even though we have to, as part of ISO13485 and FDA regulations, it's still an important part of production.
Perhaps it's not as bad as your worse case scenario - I am in the UK and received two learning kits on Monday 18th August. These would have been shipped after July 18th - and both are fine - I can programme my own sketches and programme the examples in both. Maybe I just happened to receive some that were made before July 18th and had been hanging around! Anyway, many thanks for keeping us all informed of the situation and, as the owner of a business myself, feel your pain and admire your resolve to fix it. I am very happy with my Microviews as I have been with other products I have bought from Sparkfun.
That's excellent to hear! However the ship date is only loosely associated with the July 18th firmware change. You may have had a unit that was programmed prior to the July 18th change but shipped much later in July.
Thanks for the positive words :)
Wouldn't shipping the faulty units back, fixing them, then mailing them back out be cheaper? I didn't purchase a Microview, but if I had, I would be sympathetic to your issue. I don't know if everyone is that way, but I would have tried that option first.
I do very much like your openness (willing to own up to mistakes, document what happened and what lead up to the problem, and then making things right).
Jeff
Its amazing the cost of shipping. Shipping out isn't bad, but buying return labels can be really expensive, especially from overseas. Plus don't forget then we have to pay people to accept, 2000 packages and keep track of who has sent them back and gets their replacement...
To add to this: paying to return everything and ship back fixed units is paying for two extra shipments. For many international locations the shipping alone is more than the cost of the Microview, so that really adds up fast.
Another point to consider is timing. With ~2000 returns to process it could take over a year for a majority to go through the hassle of sending off their faulty unit. By building up replacement units and shipping them out we can make it right for all ~2000 backers in a much shorter time frame.
Oh, verly likely not. The cost of labor to handle the returns and do the repairs is signficant. And the scale here would required establishing a whole new process with it's own quality control procedures -- a large distraction from dealing with the more important issue.
Excrement occurs.
As always, we appreciate SparkFun's commitment to the customer. I did not order a Microview, but if I had, I would mail the defective one back. You did the right thing, but I know I would rather you recoup some of that $58K. :)
Eh, don't feel too bad, the boards I work with can run $250,000 per board. Imagine having that in your lab and not being able to contact it.
Got an email from Sparkfun today saying that they were sending me a new MicroView because mine was in a bad batch .... except it wasn't, I can upload sketches on it just fine. I replied and said that; kinda hoping they send me another one anyway ;-)
Got an email from Sparkfun today saying that they were sending me a new MicroView because mine was in a bad batch .... except it wasn't, I can upload sketches on it just fine. I replied and said that; kinda hoping they send me another one anyway ;-)
We know for a fact that a lot of the shipped units were fine; we just don't have any way to know which ones. Unless you've explicitly asked us not to send you a replacement, we're sending you a replacement.
Happy Free Extra MicroView Day. (Month?)
Any update on the tutorial to install the bootloader on the Microview?
Yes
Thanks!
Batch 53977 - received unit missing a bootloader. Confirmed that a replacement is already in my order queue, thanks! I'm very impressed about Sparkfun and GeekAmmo's outstanding handling of this issue. Also, I'm happy as I have had an AVR programmer that I've wanted to use for a long time, and this was the perfect excuse to learn how to use it.
Cautionary Tale: The pads for Pins 11-13 are easily lifted, especially if you're using solid wire and bending it after soldering it onto the pad. I (foolishly) was hand holding a wire and overheated + moved wire = cold solder joint on Pin 13, lifting the pad right off. Using a Helping Hands to hold the wire for the other pins worked much better.
Photo of damage, plus my executed plan B for SCLK connection and AVR programmer setup is here My plan B worked just fine and I've got a functional MicroViewn now. Thank you for the bootloader install instructions!
Bravo, Nate and SparkFun. Although I wasn't affected by this, it's another example of why I love SparkFun. It would have been enough to explain that there was a problem and you were shipping replacement units, but being open and explaining what happened, and explaining how to fix the problem if you don't want to wait for a replacement... that's pretty awesome.
I hope no one got fired over this. If a SparkFun employee is behind this and not an outside vendor, please go easy on him.
It's not as important to never make a mistake as to own up to a mistake and fix as quickly as you can. Good work. Sparkfun.
I see a lot of people talking about how well Sparkfun is dealing with this screw up.
As one of the affected customers, let me review the situation:
This is NOT an acceptable solution to this problem. Credible Kickstarter performance requires that you stand by your initial agreements, and ship functional product out on time. At a minimum you need to be prioritizing the replacements ahead of people that should be later in the queue.
Oh, man! I wish I'd used the same email address for my survey response as my SparkFun account. I don't see any confirmation emails from the 11th of August, so I guess only people that had matching email addresses know their status?
So you're making batches of these every day, I should have a very early (at this point) priority. So you're telling me it will ship out by Monday?
I'm afraid probably not. There are at present about 2300 orders in our system awaiting MicroView units. A chunk of these are held up on unrelated miscellaneous shipping concerns, but of the replacement orders, yours is approximately number 1213 in line.
I just don't know enough right now to give you a concrete timeline, so I'll avoid making anything up. We're doing our best to move faster than Nate's Oct/Nov estimate from above, but we can't make any promises at the moment.
Of the ABSOLUTE queue, where do I stand in line? The only appropriate answer is 1213 in line for all orders. Not 1213 of the replacement orders, that's kind of my whole point. The replacements must be prioritized ahead of the other orders, which, pretty much by definition came in later.
I said earlier I'd avoid making stuff up. It's been a long day and I'm borrowing barely functional wireless from a coffeeshop up the street to write this, so I'm cut off from the database I usually lean on. What I'll say is that while there's not exactly an absolute queue for shipping anything here (if you want to know about this in absurd detail, there's a coworker of mine you should talk to for a couple of hours - he has a flow chart), I'd guess it's about the same as that ~1200, with some fuzz for orders that got held up for one reason or another much earlier in the fulfillment process.
As you correctly suggested originally, we're prioritizing replacements for all the potentially-busted units over subsequent rewards. I just want to be clear with everyone that there's going to be some variability in this and we're still getting a handle on the problem. The amount of patience and goodwill we've been shown in general in response to this may not be deserved, but it's greatly appreciated.
Doing some digging for the date of your order import got me thinking about dates, though. Is there any way you could check the side of the box for your bad units for a batch number? (Small white sticker with a barcode, digits below.) I'll wake up in the morning and realize I'm still wrong, but I keep vaguely thinking if I just look at it right I'll figure out a pattern.
Batch #54608.
Really not trying to bust your chops on this, but I keep getting answers that dance around the fundamental question of "Are these replacements being prioritized?" The original November date certainly didn't indicate that.
Thanks for clarifying things.
Prob should also always breakout the ISP headers too, atleast to the point where you can avoid having to pull appart the product, we're all meat bags and we all <blip> stuff up :)
But awesome work guys, alot of companies will just go "sorry your just gonna have to wait" atleast you guys provide firmware and are planning on providing detailed instructions on how to recover the board /and/ your replacing them at no cost to the user.
My batch is 54376, and it is missing the bootloader. I really appreciate what you are doing to address the problem. That's the way to keep your customers happy.
A great response by Sparkfun here, yet another reason to never doubt the integrity of your business. As it happens, I work in modern manufacturing test (trust me when I say there is a huge variation in what people consider to be 'manufacturing test'), so I'm naturally thinking of ways this could have been avoided. I don't know what kind of testing you do but in a hypothetical scenario you could have been measuring the current draw of each board to 4.5/6.5 digit precision and logging this data to a test data management/analytics system. Boards with and without bootloader may have had slightly different current draws but still within acceptable limits (or if you had really tight limits, instant failure at that point). In which case, running a capability analysis on the data would have shown a short term deviation in that particular test which would signal to someone to figure out why. The ideal solution would obviously be something that tests the presence of the bootloader directly but that is the benefit of hindsight! I don't mind admitting I work for a software test data management company but I'm here as a Sparkfun customer, so no name dropping. Get in touch if you're intrigued though.
We may be in touch. Thanks for sharing your professional insight!
Hello, my MicroView works great. No problem with the bootloader. If there is usefull in the box "batch # 53767". It was delivered on Monday, 18
Can't wait for the tutorial, mine is indeed one of the affected units. The ISP stuff no biggy, I have an STK500 and AVR Dragon at my disposal, my issue is opening the dang thing without hurting it. It looks like it was ultrasonically welded into place :-/
It's actually not that bad. The clear lens is trapped by the black plastic casing. Wedge a small tool between the two on the left or right side and the clear lens pops off pretty easily - then just push the pins from below to remove the Microview from the casing. I'm planning to install an ISP header and maybe a couple pushbuttons in mine while I've got it open...
Well, mine both work great. I guess I got lucky... or maybe not. Two more would be very welcome, they are great little units.
Anyone that has lived long enough has been tangled up in "Oh, I wish I hadn't shipped that." problems. Sparkfun is doing the right thing.
Great response from SparkFun and handling by Nathan as the CEO. I'd be curious to know how the guy who made the mistake handled this. I've seen people make mistakes and spend a huge amount of energy trying to blame everybody but themselves, and it is very destructive to the organization (though that usually isn't seen from the outside). I'm certainly a member of the $40K club (having messed up some boot code in ROM that required a new chip spin…blew right by the $40K club with that one), but by simply owning up to my mistake it went away very quickly (and I learned that despite the great lengths I went to in testing, I clearly wasn't going far enough).
Glad y'all lived, glad y'all learned, and hope to see as good or a better response (is that possible?) to the next mistake that y'all make (since like us, you're human).
Our newest inductee into the $40K club is handling this all very well and has been working his tail off the last two days trying to do everything he can to make this right, as quickly as possible, for anyone that received a bootloader-less MicroView. Everyone here at SparkFun has been so supportive. It's really incredible to see how well we all collaborate together to solve problems, especially when the pressure is turned up to 11. There's no doubt that we will come out on the other side of this a better SparkFun with greatly improved processes and an intriguing story to tell.
To anyone who receives a defective unit, and doesn't want to fix it themselves - I'm willing to buy the device off you for cheap :).
I have always thought highly of SparkFun but this just raises the bar!
I commend you and your team on the transparency and total ownership of your lesson. Looking forward to future orders and I'm even interested in learning more about Microview,
Thanks!
Apparently "Statistical Process Control" is not a phrase that's popular at Sparkfun.
What I would do instead of shipping the same unit, I'd build a programmer that allows just plug the unit in and re-flash it. Benefits: -Cheaper in production -Can be reused for other ATMega recovery -Faster lead time -Users get their devices fixed (and presumably faster) -Users get better understanding of how things work. Presumably your customers are those who are interested in such kind of things.
Other then that - you guys are great, I love you. I'd try to help your customers in northern Boston area to get their devices fully functional (I've got a few arduinos I can use as programmers) just to save you guys a few bucks.
Unfortunately they can't do that. To reprogram the AVR without the bootloader installed, you need access to the SPI port. Because the SPI port on the Microview is used to operate the display, it's not broken out on the device's pin headers. So you can't burn a bootloader to the Microview without taking it out of its case. Fortunately, it's not too difficult to remove the unit from its case. (Pretty much just pop the lens off)
Echoing the sentiments of everyone that your openness is fantastic, and keeps our loyalty. We all make mistakes, but you guys have always just made it right, and that earns lifelong loyalty from customers like me. I'll buy more not less because of this!
Also, I hope you continue to describe what you guys are doing to correct the systems of manufacturing, as this whole episode is actually quite a teaching moment to all us makers. I am hoping some of the things I am making might someday need "mid-volume" production, so learning what "the pros" do to solve potential problems in the real-world is always better than some theory. You guys are always out there teaching us stuff, so just regard this as one more thing to teach!
In the coming weeks, probably after we've tackled the whole "move to a new facility" thing, I will be writing up a post to update everyone on what improvements we've introduced to our QA/QC systems. Lots of really great ideas are already been considered. Though amidst all of the work happening to correct this big mistake, we likely won't be able to make any of the big wholesale changes until after all early adopter MicroView users are happy with what they've been shipped. Thanks for the feedback. We're glad that our documented mistakes are helping others avoid the same!
Sparkfun is awesome... I cant imagine any other company handling this the way you did.. I think the tracking says I get mine in the mail tomorrow.. I really HOPE I got bad ones because I'm looking forward to learning how to fix it.. If I do, I'll give the new replacement ones to a boy scout working on his electronics or computer merit badge so you'll have another customer for life!!
Ouch!
I'm glad you guys are making it right!
If it wasn't so easy I would say you don't necessarily need to check for the bootloader on each unit, just make sure you test the final complete firmware you lay down on a chip (burn the chip, and then go have a human test it). But since it seems really easy to add it to your automation, checking each chip/device seems like the way to go.
I don't fault people for mistakes. Even expensive ones. What matters is how you respond and you guys always do a great job. Always impressed. Once it's all settled I'll have to make sure to pick up a microview to help offset some of what you're losing fixing the problem. Keep up the solid work.
I bet there's gonna be a hot market for crippled microviews. I think fixing it up would be a good little project.
That's a shame...
This is why there is something called Verification and Validation (V&V). But I guess you live and learn.
If it's simply a matter of putting a bootloader on the device, wouldn't it be cheaper for sparkfun to send each affected customer a pocket avr programmer (https://www.sparkfun.com/products/9825) instead of a new microview? Yes, it would require each customer to flash a new .hex file but it would be faster and cheaper than sending them a second unit.
Probably, but we don't want to make a customer have to fix what we broke. We tend to have a lot of people who don't know how and just need a working unit. This is also a very unique case, because of the fact that this was a kickstarter we don't have order information making it much harder for us to contact people.
Ah, brings back those Pentium memories. Kudos for handling the situation much better than Intel did!
I feel for you -- is there a process you could do that only does extensive tests when you change the firmware - or is the "complete" testing you are doing now not impacting the production rates??
As Always, I'm really impressed with how SparkFun handles it when things go wrong. I think it's actually pretty rare for a company to be upfront and diligent in acknowledging when something unexpected happens. This is one of the key reasons SparkFun continues to be the First place I go to look for information, products, components, etc. You folks so way Rock!!
Will there be an option to simply refund/cancel the order? Frankly November is a significant change from the early summer timeline that I funded.
That's a reasonable question that I don't have an immediate answer to. If this was a regular product we'd absolutely work with you to cancel the order and refund your money. Since all the transactions went through Kickstarter and Geek Ammo we have to work through them. Shoot an email to microview@sparkfun.com and we'll get it worked out.
Bummer! I'm sorry to hear that it's going to cost so much, but again glad to hear that SparkFun is such a good company. I have been referring friends and coworkers to you in the past, and will continue to do so in the future. :)
As for the MicroView, I think I received one of the bogus ones, but in Arduino 1.0.5-r2 I'm seeing a slightly different error than one of those listed above: "avrdude: stk500_getsync(): not in sync: resp=0x00"
I love the Look of the Microview and how the Demo on it runs, My son had fun with it. This makes sense, I was having problems Uploading a Sketch. how hard would it be to put an ISP footprint on the Bottom of the MicroView?
Wow, that sucks. I seriously hope that most people will just try to fix their own bootloader. I know I definitely will. The last thing I would want is to cost you guys money by returning it. You guys are AWESOME.
Thanks Mike! And we greatly encourage folks to try to fix their own bootloader. To clarify, you don't need to send anything back. We are sending a 2nd replacement unit. To those affected.