For historical reasons, the current code base on the soon to be retired version 1 platform is a mix of C and C++ in the Arduino framework, using the Arduino IDE, which was creaking due to what the firmware had evolved into. Change is necessary, so lets look at the options and where we go next.
In previous roles and projects, I have worked on variety of low level solutions using a number of different microcontrollers, microprocessors, including 6502, 6809, Ubicom (Scenic) SX48, various Microchip PIC's, Atmel AVR's and a collection of programmable logic devices. So, I am experienced in assembly language, C and C++ amongst other languages.
The timing of the Chicken Coop v1, the arrival of the Arduino framework, plus my children being in the start of secondary school when the project started, and me not wanting to drop them in at the deep end on learning to code, meant that the Arduino framework was worth a look, particularly since it had a good collection of libraries for things I wanted to use and as anyone knows, who has stated with an empty editor at the start of an embedded project, writing good drivers takes a lot of time.
As a result, the initial code base is a mix of C and C++, with a collection of standard libraries and a number of my own. This allowed a quick "wow factor" when the family saw what we could do with the new computerised chicken coop.
However, as time passed any my son started to go down the Electronics route and head off to College and then University, the need to use the Arduino framework decreased and became a comedy point at home when he became competent in professional grade tools and frameworks. However, there is a stable code base that I want to re-use because of the time pressure to replace the old hardware before it fails completely.
Upgrading the development tools and re-writing code can be done after the hardware build is done and indeed that makes things a lot simpler, since you then know that the hardware is good, rather than not knowing if any specific problem you are looking at is hardware problem or a software bug.
The Arduino IDE is targeted at beginners, with relatively simple needs and the size of the code base for what the v1 project was stretching the simple IDE and I was hitting some limitations:
- The lack of alternative download methods that are faster. We are stuck with bootloaders and UARTs to perform the download.
- The build process is really slow, rebuilding far too much that hasn't changed each time. If only there was a Makefile
- No JTAG support, so debugging is very laborious with Serial.print() and wiggling pins to indicate presence in certain routines.
- The Arduino IDE has only basic capabilities
- It does not cope well when you have multiple modules as it opens new tabs at the top of the screen, so lots of scrolling and dropdowns had to be used
- The search and replace tools are simple
- There is no real context based info on functions
- There is no source control integration, which required manual processes to be used to protect the code
- There was no easy way to flip between development and production settings
- My hacks to Arduino build scripts to allow programming of the production coop in the garden, gets broken each time the IDE gets updated.
How to fix this ?
The reimplementation of the system provides the opportunity to refresh the software stack, however, the amount of work to do between now, at the design stage and the desire to install in the spring / summer of 2023, only 5 months away, is already a difficult challenge and as mentioned above, re-writing existing code in a new development framework on new and fairly complex boards will complicate the work significantly. Remember that this is supposed to be a fun project, not a day job. My current codebase is relatively bug free, so this will be the springboard that version 2 starts with.
Around the same time, that I started thinking about the new revision of the project, I was getting involved in 3D printing and that quickly ends up in the Marlin, the open source 3D printer firmware which uses Platform IO and Visual Studio Code (VSCode) if you want to recompile it. Very soon after that, I started looking at ESP32 microcontrollers from Espressif, who also use VSCode with their Integrated Development Framework, the ESP-IDF, since that would allow me to move away from Arduino.
Having looked at the various overlapping tools listed above and not really understanding where one finished and the next one started, things got a bit confusing, until you peel back the layers and start to understand it a bit more. I also found that PlatformIO could wrap around the Arduino framework, but at that time, it was quite rough around the edges as it had only recently been made available.
As luck would have it, a friend needed a bit of help with another relatively simple ESP32 project, so some practical experience in the above tooling could be obtained and the penny dropped into place on each of the components. The Arduino project also shipped a new IDE, which is clearly a cut-down version of Visual Studio Code since the menus and window layouts are the same, but this was also work in progress, but showed that the Arduino project had recognised the limitations in their own IDE.
OK, so what about version 2?
Based on the above information, the tooling available today and current time pressures, the approach for software development will be :
- Use Visual Studio Code will be used as the IDE, since
- Its a more modern and mature IDE.
- Its far better at managing larger projects
- It has much more modern features including showing function prototypes
- It provides Doxygen documentation for functions when you hover over them.
This is dependant on the correct documentation structure being present in the source code. - It uses a proper folder structures for source code, include files and libraries, documentation
- it has a much faster build time as its intelligent on what's changed.
- It can get you to a function prototype or function implementation in one keystroke or mouse click
- it provides GIT integration in a nice easy to use, visual manner for all the common tasks (committing, diffing changes, looking at the branch history, etc).
- You can document things in Markdown, (the opposite of Markup / professional typesetting)
- A local GIT server will be used for source control
This is not an open source project, and the code is not on GitHub - PlatformIO with the Arduino framework
Initially for phase 1 and initial testing on the new boards.
This approach also allows access to a percentage of ESP-IDF functions, which simplifies transition to that framework. - Then, transition the project fully to Espressif IDF.
Other project report that such a move frees up a lot of resources on the ESP32.- Migration will start on some of the simpler modules first to gain exposure to that framework.
- The principle of separating configuration from code will be used
- Secrets will be stored in the ESP32's Preferences storage and not in source code.
- This also allows calibration data to be stored on the ESP32, since that will be unique to each board and the tolerances of the components fitted to it.
- Utilise native features of the ESP32 family, such as the real-time operating system, FreeRTOS This allows tasks to be isolated, messages to be sent, interrupts to be handled, hence a better use hardware resources and cleaner code.
- Remove any obsolete coding styles / requirements for the Atmel processors from the code base, such as PROGMEM / F() functions, which move static strings into Program Memory (PROGMEM) to conserve the very limited RAM (8K on the Mega 2560). This is not required on the ESP32, which already puts static strings in FLASH memory, hence such functions are simply ignored and they can be removed.
- Add Doxygen documentation in for each function. This will mean going back through all the code, but also provides a good opportunity to clean any obsolete code.
- Use the DRY principle so that common code between any of the separate system modules is exposed as a library when possible.
- Note that support for this in VSCode and PlatformIO seems to be imperfect at the moment.
- Key device classes(PWM, ADC, IO, etc) will be exposed via C++ classes, this allows for all of one family of device to be managed by one module and for references to channels of that object type to be passed to other functions.
- The benefit here is in uniformity for the calling method in other functions, higher level components will need access to the lower level functions (ADC channels, PWM outputs, GPIO pins, etc), by using a class that can cope with all ADC types simplifies other code and makes it trivially simple to add new hardware devices in, or to replace obsolete devices over time.
- The module can cope with the technicalities of how they exist on the board - are native to the ESP32, are on any I2C, SPI bus, etc. Yet, all higher level functions just see a logical channel handle. This greatly simplifies the codebase for higher level module types particularly and it simplifies things when there are multiple instances of a device within the system, so several IO Expanders, several ADC devices, etc.
- When combined with the single instance of libraries across different boards, this makes the code a lot more flexible and simplifies swapping IC types over time.
It will clearly take some time to get to this endpoint, but that is what goals are all about