Arduino Trick: Double Upload Speed
A simple non-spammy trick to double the speed you upload to your Arduino board.
"Double your upload speed" sounds like a spammy internet site. There is however a really simple way to double the speed you upload to your Arduino.
Deselect Verify code after upload in the preferences window and click ok. That's it. Don't believe me? Open this big sketch that takes up ~30k of flash (nearly all the flash on a standard Uno). With the box checked it will take about 24 seconds to upload the file. With the box unchecked it will take about 13 seconds.
What's going on here? By default the Arduino IDE verifies that everything was written correctly:
Program Step:
Arduino IDE: Hey
Uno: Oh hi
Arduino IDE: I've got some new code for you
Uno: Great! Send it to me
Arduino IDE: Here it is... [30k of bytes]
Uno: Got it, thanks!
Verify Step:
Arduino IDE: Hey
Uno: Oh hi
Arduino IDE: I'm not sure I trust you got everything correctly. Send your flash to me.
Uno: Ok, here it is... [30k of bytes]
Arduino IDE: [Compares Arduino bytes to original bytes] Hmm, looks ok. Please proceed.
This is how almost all programming routines work: send the code then verify if there were any errors during transmission. By skipping the verification step you reduce the number of bytes that have to be passed back and forth by half. What you may not realize is that there are at least two other error checks: one at the bootloader level (each frame of the STK500 bootloader has a cyclic redundancy check) and even lower at the USB to serial communication level (each USB frame has a CRC). If any communication gets corrupted it is caught and corrected at these levels. It is highly unlikely1 that your sketch will be recorded incorrectly to the Arduino when you hit Upload.
Why does this matter?
Many sketches are a few thousand bytes so turning off verification will only save you a few seconds per upload. But how many times do you upload a sketch when you're working on a project? 10 times? 50? It's more than you might like to admit. And how many projects might you work on? I've probably uploaded tens of thousands of sketches over the past few years. Now multiply by all the Arduino users out there and you end up with a tremendous amount of wasted time.
Nathan: 25 sketch uploads * 100 days a year * 6 years = 15,000 uploads
Time wasted: Avg time savings of 5 seconds per upload * 15,000 uploads = 75,000s = 1250min = 20 hours
Arduino users (wild guess): 3,000,000 * 20 hours = 60,000,000 hrs = 2.5m days = 6,849 years of wasted time
These numbers are obviously unscientific but you get the idea. We'll all be better off by spending less time watching the TX and RX LEDs blink.
Times when you might want verification
What a failed verification looks like in Arduino IDE
There are times when you may want to verify your code. If you're going to deploy your Arduino into a satellite or into a final project you may sleep better knowing the code is correct. If you've got an extraordinary connection to an Arduino like a 50ft USB cable or a 2km connection over RS485 you may want to verify after upload. It's still unlikely an error will slip through the CRCs so use your own judgement.
What boards does this trick work on?
This works with any Arduino that uses a serial to USB IC (Uno, Pro Mini, LilyPad Simple, Fio, etc). These boards all use the same avrdude bootloader that use the verification flag by default.
Any board using the Catarina booloader (Leonardo, Micro, etc) or the Halfkay bootloader (Teensy boards) have much faster bootloaders that don't see much, if any speed advantage.
[1] It's perhaps better than 1 in a million but I'm not sure how to calculate the odds so please let me know if it has been. In 11 years of hammering on microcontrollers with serial bootloaders I've never seen an incorrect record to flash. Any firmware errors were always because of my own fault or faulty bootloader design.
Save time, trust your toolchain, and uncheck the box!
I don't think some of the people below read the article. A CRC checksum is performed after upload which is a pretty good verification that the code integrity is correct.
I recommend turning off verify code after uploads during development for performance and then turning it on for your production release.
Thanks for the article.
That is not correct - CRC is done only by USB and the bootloader, verifying that each packet of the data is not corrupted. Which won't protect you from e.g. failing flash. There is no "CRC after upload" and the article never said anything like that neither.
The post-upload verification works by downloading the flash and comparing byte by byte with the original hex file. If it doesn't match, verification error is raised. Of course, that works only if you keep that "verify" switch on.
Quoting the article
"each frame of the STK500 bootloader has a cyclic redundancy check"
I don't know if this is true or not, but I know that it is NOT correct that "the article never said anything like that..."
Having a bootloader without some sort of checksum or CRC is unlikely, and it would probably exist at the record level rather than at the file level.
Uh, no. The UNO R3 bootloader, 'Optiboot', performs NO CRC CHECKING WHATSOEVER. It handles only framing errors.
Not the point. The point was whether or not the article SAID it did. The article DOES say that it does. Nowhere was Optiboot mentioned. Maybe Optiboot doesn't, don't know, but again, that never was the point.
Also, quoting the Omimex manual for Sparkfuns STK500 Compatible USB Programmer: "Boot loader implements the standard protocol XMODEM with CRC16 for firmware update." The Atmel manual states that it uses a checksum on each record sent, both are options for the Xmodem protocol. Completely ignoring the checksum/CRC response is also an option, but not recommended.
Any competent Xmodem host will resend a record if the client responds that the checksum/CRC failed. Program downloaders have been routinely doing this since the '70s. If Optiboot doesn't, it certainly should!
I enjoy the 13 seconds of down time anyway. I usually use it to contemplate the many things I coded incorrectly or the wire connections in doubt.
What a wonderful idea ... Save a few seconds on verification and then waste hours debugging incorrectly uploaded code, chips with failing or locked/not blank flash (USB nor bootloader CRC won't protect you from that!), corrupted uploads due to intermittent or noisy connections, etc. I have seen avrdude fail in all kinds of wonderful ways, even when using bootloaders.
Really "worth" doing. facepalm
As an embedded programmer with over 30 years of professional experience, I can tell you that my experience is that a code download failure is very rare, and generally catastrophic: the very few times I've had a failure, the processor tends to immediately crash hard, or crash whenever a certain operation happens. In my experience, it doesn't really cause the kind of flaky behavior that would waste hours of debugging time.
In the early days, when using EPROMS that had to be erased by ultraviolet light, I would always do a verification as those chips were temperamental. But that wasn't so bad because the entire image would be downloaded into the programmer (a slow process at 9600 baud) then it programmed the chips (an even slower process) and the actual verification went fast because it just compared the chip to the internal memory image and didn't involve a new download.
But in a modern development environment, the serial download speed is much faster, and the flash programming is much faster and more reliable, that the verification process takes a much larger percentage of time. It's not so much of an issue with Arduino where even a large sketch takes just a few seconds, but when dealing with a larger flash space and a lot of code (1 megabyte of internal flash on my current project) the verification time can be significant - long enough to go to the men's room and return with a cold can of caffeine and have it still not be done when you get back.
I turn verification off during development. But I always leave it on in a production environment (and add checksums to the code that get verified on every boot.)
Er... most of that would get caught by the CRC checksum.
Bootloader CRC won't protect you when the chip's flash is not erased or failing. It protects you only from data corruption in transfer (e.g. due to noise). And even that isn't infallible - the checksumming is usually fairly rudimentary.
My question is why would anyone not want to verify that the data is correct after programming? It's a smalls thing to do to make sure that the chip itself is still (for the most part) working. It allows you to identify issues along the toolchain if it doesn't match.
In the end, I'd rather be safe than sorry.
I agree for the typical Arduino sketch, it's a small price to pay, even though the odds of a failure are very remote. For larger projects on bigger processors, it can be a significant amount of time. Long ago, I turned off verification on the 1MB downloads I routinely perform for work - it's saved me LOTS of time. For play time on the Arduino, I don't know if it's worth the effort of making the change.
The reason you want to verify your code upload is because flash memory eventually wears out. If you go through a lot of upload cycles during development you will eventually find the limit where your flash memory is suddenly read-only. Even worse, you're overwriting the same locations with every upload - there is nothing to make it so you wear every byte of flash evenly like there is in an SSD.
I know I can go through a thousand uploads or more during the course of a year and I only own a couple of full size arduinos. Generally my work moves to a pro mini version once development is complete.
If you use a virtual environment, like I did, disabling the verification is not a good idea. I have several Unos and Megas and they ALL had problems with USB to a VMWare Workstation 8, Win 7, 64bit set up. After a few months, I finally talked by boss into giving me a real 2nd computer and that solved the upload error problems.
This tip is valid for serial programming, but I do not recommend this if you're programming via ISP. The SPI programming interface is not checksummed.
Note to self (publicly)... never trust anything from sparkfun if it is going to be part of something that could injure or kill a human.
...and definitely don't go mountain climbing, flying, parachuting, or any other extreme activity with someone from sparkfun that has had a hand in building, operating, or otherwise touched any gear used in said extreme sports without double checking the work they have done.
Don't worry - we only test our really dangerous stuff on internal folks.
Thanks for this simple tip. I'm currently developing code on an Arduino Mega 2560 currently in excess of 50K. Your tip will save me hours.
In my experience, your code is more likely to be corrupted by a "brown out" condition (along with the bootloader).
Funny, I have never had "verify code after upload" enabled and haven't had an issue since I installed the IDE.
I think this is a great post (comments included), not in the fact that people will begin to bypass the flash verification; but this should motivate those involved into learning how the bootloader works. By the way, you should be able to browse the bootloader source code to figure it out.
Quoting the article: "It’s still unlikely an error will slip through the CRCs..."
Try to figure out if the bootloader actually performs Cyclic Redundancy Checks (CRCs) on the written flash vs. data to be written to flash. I don't know, but I don't think it would perform it on the written flash...but let's just find out for ourselves?
After checking 'STK500boot.c' - There are no CRCs performed...I did notice an xor checksum routine which the bootloader does perform on received packets. Keep in mind, XOR checksums are not very robust and they are NOT CRCs (CRCs are much more robust, but they are not perfect).
The Optiboot bootloader doesn't seem to perform any checksums/crcs at all...
Maybe not verifying correct flash upload is not such a great idea (on paper)?
Checking a few instruction manuals, different Host/Client combinations seem to use all the options from full CRC, to XOR checksum, to completely ignoring the checksum/CRC altogether.
The way the protocol is supposed to work is that the host will inquire if the client supports CRCs, if so then it uses them. If not it is supposed to fall back to a Checksum. Even with a checksum, there is no law saying that the host needs to pay attention, so apparently some don't. The standard way of responding to a bad CRC/Checksum response is to resend the record. This is much quicker than resending the whole file like the "Verify" flag does.
Of course this is quicker, thus the article. However, the point to be made is that the actual programmed flash is not being verified at any level of the bootloader. Any failed-write (which is absolutely possible) will go unnoticed, because it is not being "Verified" by the verification option. Don't get me wrong, they put that check-box in there for a reason right? (don't want to verify your upload? Live on the edge and uncheck the box :0p
I think the bigger picture here is what kind of embedded programmers to promote; those who get in the habit of cutting corners to save literally only seconds per upload, or those who can understand and appreciate the pitfalls of memory failures and robust verification processes to combat them.
IMHO this would work well on a production line, as it would be faster to flash, and then if you have a test jig you just test it as normal, and try reflashing w/ verification if it fails. If it's still bad after reflashing, then something else is the problem :)
Possibly the silliest thing I've seen come out of www.sparkfun.com
I submit that you probably haven't been paying that much attention. Silly is pretty much our MO.