Our customers a decade ago and now always wanted to update their embedded SOM products with latest software features, bug fixes and security patches. With the explosion of mobile technologies, internet speeds and devices in everything it is a more pronounced requirement today when our SOM customers ship their products. Driver-less cars and automotive segment in general are sending ripples around the internet for adopting firmware upgrade in their offerings.
Part – I Requirements
Of these thousands of firmware update queries, we have received from our customers, below we try to make a general classification. We hope this classification will give the reader an idea on – development trade-offs, field maintenance, system design and deployment challenges in bringing firmware upgrade solution on a device. Firmware update requirements can be broadly categorized as follows:
1. Legacy – “I have this legacy firmware upgrade script, when I run it from my jump drive it replaces all the binaries on my device. It does some primitive checks before replacing. Can we have it on this new generation device with backward compatibility?”
2. Cloudy – “I have this legacy firmware upgrade mechanism, can we migrate it to cloud. What are my cloud options?”
3. Security – “I have this non-secure firmware upgrade process, which is very brittle on my customers hands, can you fix it?”
4. Downgrade – “My case is special my customers are looking for a downgrade option for some devices in the field.”
5. Network – “My devices run on a low bandwidth connection, my legacy upgrade is time consuming to download and install.”
6. Dead brick – “I am looking for a robust firmware update solution where the device will try to update successfully, if it fails, it should be in a state to run the previous version.”
7. Memory – “In my system design I do not have a lot of extra memory that is required for firmware upgrade, can I get a small footprint firmware upgrade solution on my device.”
8. NextGen – “I am designing my next gen device platform, I am looking for a robust, secure and scalable firmware upgrade system in place. What are my options?”
Overall one can infer that system designers have a broad spectrum of requirements starting from systems trying to keep up with primitive legacy solutions in modern day to systems trying to leverage latest technology advancements. Apparently, every system designer is desirous of firmware upgrades for their long lasting devices. Planning, designing and developing firmware upgrade solutions is a different game altogether. The rest of this paper explains nitty-gritty of designing and implementing a firmware upgrade solution.
Part – 2 Challenges
In principle a firmware upgrade is all about moving a desired set of files on to a device and changing some configuration parameters for run-time requirements. In the olden days people saw this simple principle and created simple scripts with copy and edit commands for their devices. As systems demanded upgrade and downgrade features the scripts started growing in complexity. Over time the electronics became cheaper, the number of devices in the field grew demanding for centralized monitoring and control of firmware versions on devices. This presents a whole new set of complexities and challenges for firmware upgrade solutions of modern day. Let’s analyze some firmware upgrade system requirements and their challenges.
Case – 1: Upgrade or downgrade device firmware
An upgrade or downgrade in first place needs either extra storage or a reachable firmware repository to pick up the bits. When upgrading either because of issues in infrastructure, power or memory the process may stall. The process can run in the foreground or background and can be automatic or manual. In all cases the system is expected to return to a usable state as quickly as possible.
Case – 2: Upgrading with integrity
When downloading a package for firmware upgrade the package may not be entirely available because of a communication loss, repository unavailability or memory issues. In all these cases the firmware upgrade solution should be able to perform integrity checks before the device can switch to this newly loaded package.
Case -2: Trusting package sources
In centralized and decentralized firmware upgrade solutions the trust on the origin of packages becomes a question. Any intruder can pose as the creator in what the devices are looking for and compromise a system. Man-in the middle attacks, mirroring attacks, forever downloads, non-secure communication channel hacks are common threats the firmware upgrade solution need to address in this implementation.
Case – 3: Security concerns
Security issues arise at different levels when building a device firmware upgrade. Security at the device level, security at the repo level, security at the developer bay. At the device level we need to establish highly secure channels with trusted repositories. Even human intruders should be kept at bay by implementing privileges and pass-codes to change any firmware repo access on devices. In practice for device repo communications industrial strength protocols such as https and secure web-sockets are considered.
At the repo level, we cannot keep our communications open to anyone requesting for firmware packages or root meta information. Repo’s should be self-corrective and be able to distinguish between genuine mass upgrades and DoS attacks. Trusted mirrors must be distinguishable and should follow standard practices for updating repositories. Various signing procedures should be followed for root metadata, package metadata and packages by themselves. When updating metadata and during consumption mechanisms such as date of expiry should be implemented to safeguard devices from repo attackers.
At the developer level they should follow best practices to sign their binaries with restricted access private keys, using project and developer keys. They should employ quick communication mechanisms to inform devices in case of compromised keys and repositories.
Case – 4: Bandwidth issues
Many times, devices are grouped together in a private intranet or cloud with bandwidth constraints. Some devices may even exist on legacy networks waiting for upgrades. In all these cases sending out huge chunks of bits can create network problems. Firmware upgrade in these scenarios may employ different mechanisms such as: download a copy to the local network server instead of each device pulling the same copy from outside network, some repos will serve only the difference bits the devices will patch it up, in some cases a device will pull the new firmware and serve the bits to its peers. Peer dependent solutions pose availability issues and create dependency problems.
Case – 5 Partition challenges
Some system designers have budget for extra partition to store backup firmware others don’t. When memory is at premium they fallback to network repo support and binary size. Having that extra partition for firmware upgrade should be planned from day one of the system design. The logic to detect and fallback should be part of the run-time logic of the firmware update solution. Unless the system is large the firmware upgrade binary is expected to have a smaller footprint on – storage, and run-time memory on the device. Unfortunately, it’s given secondary importance till the first version of the product reaches the field, which is not supposed to be the case.
Case – 6 Upgrade monitoring
No article these days can go on without talking about Internet of things. Techniques like IPV6 makes more sense in relevance to IoT. As we move more and more devices from our factories to the field, upgrading them needs more and more control. In medium and large-scale deployments we use analytic tools at various levels for firmware upgrades. It becomes mandatory to know which systems are vulnerable because they are running a particular version of firmware. Which devices are on what products, how many times has a device attempted firmware update, did a device ever come online after deployment, with firmware upgrade one can patch devices in California differently from devices deployed in New York and analyze user behavior, etc.,
It will be termed an overstatement if we claim these are the total requirements and challenges of a firmware upgrade solution. But these are the major requirements and design challenges one has to make when building a robust, reliable and secure firmware upgrade solution.
e-con engineering teams have helped several of its SOM customers to evolve from their self-made solutions over a decade. e-con being the very early adopters of cloud and mobile technologies has a need to address the various pain-points and pit-falls of such home grown techniques. Its passion to help customers to bring up new products to market has made e-con to research and develop solutions that reduce the time to market.
Our flexible and advanced cloud based firmware upgrade solution for all your problems is coming very soon…
eSOM++ Team