Contributing to the Arduino IDE

The great thing about open source software is that when you want a new feature, you can just add it!

by Nick Poole October 14, 2020 4:22 pm UTC 3

• Arduino
• Programming

Favorited Favorite 0

Something I love about my job is that I never know what I'm going to be working on next. Anything from hardware layout to mechanical design, to firmware and software, everything is fair game. If I don't know how to do something, I'll try to pick it up. This has led me to learning bits and pieces of a lot of different skills. All of this is to say, when Nate asked me if I could add a new feature to the Arduino IDE, my first thought was, "Well, it's been a good run but nobody avoids Java forever."

A Lot of Legacy

Jokes aside, I was actually a little apprehensive. Not so much because of the Java — although it's never that much fun to install a new toolchain — but because the idea of contributing to such a large, ongoing project can be intimidating. The Arduino GitHub Repository contains over 7,000 commits in 87 releases over the span of 15 years... that's a lot of legacy! Somehow it's managed not to turn into a massive pile of spaghetti over that amount of time, which is a testament to the community around it. I'll get into the process around pull requests and code review here in a minute, but first let's talk about the new feature: Recently Used Boards!

One side effect of Arduino's tremendous success over the years is that the ecosystem of Arduino-compatible development boards has absolutely exploded. Combined with the addition of the Boards Manager, it's easier than ever to cram your "boards" menu with hundreds of board packages. For the average user this probably never becomes annoying, but in our line of work at SparkFun — constantly switching between development boards — we end up doing a lot of scrolling. The most recent version made small steps toward fixing this issue by splitting the boards into submenus. That definitely helps, but depending on your monitor resolution, scale factor, etc., a lot of boards still can't be located at a glance. Nate's idea for solving this? Do the same thing that we already do with sketches and keep track of the most recently used items in an easily accessible list.

Implementation

I won't get too into the process of installing and configuring the Java toolchain. It's no more or less painful than any other development package. That said, one thing did trip me up for about a day so I feel obliged to mention it here: You cannot build Arduino with the 64-bit JDK, you have to use the 32-bit JDK (as of 1.8.13 anyway, I imagine this will change in the future).

So, anyway, how do I go about adding a new feature to an unfamiliar code base in an unfamiliar language? Step one for me is to basically mock it up. Find the menu structure in the source code — by searching the whole repo for the string "Tools" and then drilling down until I see "boards" — and add some menu items that look like what I want, even though they don't do anything when you click them. Once I know how menu items are represented in code, I can find out what actually happens when you click them and how to insert them into the menu programmatically.

So how do the board menus work in the Arduino IDE?

Basically all of the menus are periodically rebuilt at different points while the program is running - some on startup, some whenever certain pop-ups are opened, some after hitting compile, etc. All of this happens in app/src/processing/app/Base.java. The boards menu is rebuilt by a routine appropriately named rebuildBoardsMenu() first on startup and then again whenever the Boards Manager or Library Manager is opened. It was easy enough to add some menu items here for the mockup and it looked pretty good! So now I just had to solve two problems: 1) They were static and didn't actually reflect any sort of board history, and 2) they had no actions assigned to them, so clicking them did nothing.

To keep track of the board history I would need unique identifiers for each board and a place to store the list that would persist between sessions. The unique ID was easy, since these already exist as part of the board's profile and is used to refer to the board while building the menus. I can simply query the ID of a board whenever it's selected and save it to the "recent boards list" for reference when building the menu later. This list would be stored in the one file that holds all persistent data in the IDE: preferences.txt

The preferences file is interesting; it gets loaded into an object at the beginning of each session where a number of methods allow it to be read and modified and then at the end of a session, it's serialized back into the text file. Methods are provided for storing and retrieving strings and collections by arbitrary key names. Anything that persists between sessions or across windows is stored here, so things like the selected COM port, recent sketches and, of course, preferences. In fact, if you've been using Arduino for a while, you might want to check in on your preferences.txt file because while I was playing with it I found that the recent.sketches field doesn't seem to be managed in any way and is simply a record of ALL sketches that have been opened since installation (and therefore will grow indefinitely).

To keep track of the five most recently used boards, I found a function that gets called any time a board is selected (onBoardOrPortChange()) and added a short chunk of code to capture the selected board's ID. Then I check to see if preferences has a key called boards.recent. If it does, then I grab that list and check whether the current board is on it. If it is then I move it to the top of the list, if not then I append it to the top of the list. The list is then limited to five items and stored back into boards.recent.

Whenever the boards menu is rebuilt, so are all of the actions assigned to each menu item, as well as the custom menus assigned to each board (things like bootloader and processor selection submenus), which makes that a good time to grab these actions and copy them into new menu items. Unfortunately, this isn't a great time to build the "recent boardS" menu because — again — this only happens when the Boards Manager or Library Manager is opened and closed. We'll need to store these actions and then use them to build the menu later. Originally, the list containing these actions wasn't scoped outside of createBoardMenusAndCustomMenus() so I had to widen the scope of this and a few other variables. With this list in memory, however, we now have freedom to build our menu whenever we need to.

Back to our friend onBoardOrPortChange(), and we write a little chunk of code to create the new menu items and insert them into the menu... but wait, where in the menu? We can't hard code the menu index, because certain menu items will come and go depending on preferences so our target index is liable to change, but it will only change whenever the greater 'boards' menu is rebuilt, so in rebuildBoardsMenu() we pick a spot for our menu in line with the rest of the menu rebuild and take note of the index at that time.

TL;DR

• During rebuildBoardsMenu(), check to see whether preferences has recent.boards. If so, insert separators and a menu label to denote the "Recently Used Boards" list and store the index of this section for the insertion of menu items later.
• During createBoardMenusAndCustomMenus(...) create a collection of 'clone' menu items whose action is to click the original menu items. Reference them by boardId.
• During onBoardOrPortChange() update the list of boardIds in recent.boards. This list is treated as a stack with new boards popping off the oldest member of the list. Then call rebuildRecentBoardsList() to remove previous menu items and insert new menu items using the collection generated during createBoardMenusAndCustomMenus(...).

Contributing my Changes

Arduino's development policy is listed on the wiki and is a great place to start if you're thinking about joining the development effort or proposing changes. Once you've decided you're serious about contributing, check out the CONTRIBUTING.md file for contributing rules and guidance on where to contribute to different efforts. Pay special attention to the section on Pull Requests, which outlines the requirements for making a useful PR. Most of these are good git-manners in general, and help to prevent the project becoming unsustainable under the weight of a thousand devs. Things like writing good commits, removing commented-out code, and addressing one bug fix or enhancement at a time help developers understand what you're trying to achieve and make code review go smoothly.

One key step that I neglected was to search the existing PRs to make sure I wasn't reinventing the wheel. There turned out to have been a PR at some point that addressed this idea but it lacked persistence between sessions, so I got lucky.

Once I forked the Arduino repo, made my changes in a new branch, and filed my pull request, a number of automated tests kicked off to ensure that the proposed code was still buildable and properly linted. Then I just waited for someone to review my code. Before long, I had a number of suggested tweaks in the comments and that's where it currently sits. I'm still making the proposed changes, but there seems to be interest in pulling the code, so hopefully you'll be able to find this feature in the next official release. In the meantime, if you want to take it for a test drive, you can download this file to replace the current 'pde.jar' in your arduino/lib/ directory.

Moral of the Story

Don't be afraid to contribute to big open source projects! The only way these tools get better for everyone is if everyone pitches in and helps. Have a great idea but don't have the chops to implement? Talk to folks in the forums - someone who has the skills might like your idea and help you out with the proposal! Hopefully, you'll all find this minor enhancement useful and I cannot wait to see what idea you have for the Arduino IDE.

Happy hacking!