Skip to content

Commit

Permalink
Added deployer tool.
Browse files Browse the repository at this point in the history
  • Loading branch information
slav-at-attachix committed Mar 23, 2021
1 parent 237fa66 commit 0a9b28a
Show file tree
Hide file tree
Showing 11 changed files with 529 additions and 6 deletions.
78 changes: 74 additions & 4 deletions Sming/Libraries/OtaUpgradeMqtt/README.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,63 @@
OTA Firmware Upgrade via MQTT
=============================

.. highlight:: bash
.. highlight:: c++

Introduction
------------

This library allows Sming applications to upgrade their firmware Over-The-Air (OTA) using the MQTT protocol.
MTQTT has less overhead compared to HTTP and can be used for faster delivery of application updates.

Using
-----

1. Add ``COMPONENT_DEPENDS += OtaUpgradeMqtt`` to your application componenent.mk file.
2. Add these lines to your application::

#include <OtaUpgrade/Mqtt/RbootPayloadParser.h>

#if ENABLE_OTA_ADVANCED
#include <OtaUpgrade/Mqtt/AdvancedPayloadParser.h>
#endif

MqttClient mqtt;

// Call when IP address has been obtained
void onIp(IpAddress ip, IpAddress mask, IpAddress gateway)
{
// ...

mqtt.connect(Url(MQTT_URL), "sming");

#if ENABLE_OTA_ADVANCED
/*
* The advanced parser suppors all firmware upgrades supported by the `OtaUpgrade` library.
* `OtaUpgrade` library provides firmware signing, firmware encryption and so on.
*/
auto parser = new OtaUpgrade::Mqtt::AdvancedPayloadParser(APP_VERSION_PATCH);
#else
/*
* The command below uses class that stores the firmware directly
* using RbootOutputStream on a location provided by us
*/
auto parser = new OtaUpgrade::Mqtt::RbootPayloadParser(part, APP_VERSION_PATCH);
#endif

mqtt.setPayloadParser([parser]
(MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer, int length) -> int
{
return parser->parse(state, message, buffer, length);
});

String updateTopic = "/app/test/u/4.3";
mqtt.subscribe(updateTopic);

// ...
}

See the :sample:`Upgrade` sample application.

Versioning Principles
---------------------
To simplify the OTA process we strongly recommend the following versioning principles for your application:
Expand All @@ -29,11 +78,11 @@ Theory Of Operation
Depending on the size of the new firmware and the speed of the connection an update can take 10 to 20 seconds.

2. The application connects via MQTT to a remote server and subscribes to a special topic. The topic is based on the
application id and its current version. If the current application version is 4.3.1 then the topic that will be used for OTA is ``/a/test/u/4.3``.
application id and its current version. If the current application id is ``test`` and version is ``4.3.1`` then the topic that will be used for OTA is ``a/test/u/4.3``.

3. If there is a need to support both stable and unstable/nightly builds then the topic name can have `s` or `u` suffix. For example
all stable versions should be published and downloaded from the topic ``/a/test/u/4.3/s``. For the unstable ones we can use the topic ``/a/test/u/4.3/u``.
If an application is interested in both then it can subscribe using the following pattern ``/a/test/u/4.3/+``.
all stable versions should be published and downloaded from the topic ``a/test/u/4.3/s``. For the unstable ones we can use the topic ``a/test/u/4.3/u``.
If an application is interested in both stable and unstable versions then it can subscribe using the following pattern ``a/test/u/4.3/+``.

4. The application is waiting for new firmware. When the application is on battery than it makes sense to wait for a limited time and if there is no
message coming back to disconnect.
Expand All @@ -51,6 +100,27 @@ One MQTT message contains:
Based on the :envvar:`ENABLE_OTA_VARINT_VERSION` the patch version can be encoded either using one byte or a `varint <https://developers.google.com/protocol-buffers/docs/encoding#varints>`_.
Based on :envvar:`ENABLE_OTA_ADVANCED` the firmware data can be either without any encoding or be signed and encrypted.

To simplify the packaging this library comes with a tool called ``deployer``. To create a package type the following from your application::

make ota-pack OTA_PATCH_VERSION=127

Replace 127 with the desired patch version.
If the option OTA_PATCH_VERSION is omitted from the command line then the patch version will be generated automatically and it will contain the current unix timestamp.

Once a package is created it can be deployed to the firmware MQTT server using the command below::

make ota-deploy MQTT_FIRMWARE_URL=mqtt://relser:relpassword@attachix.com/a/test/4.3

The MQTT_FIRMWARE_URL above specifies that

- protocol is: mqtt without SSL.
- user is: relser
- password is: relpassword
- host is: attachix.com
- path is: /a/test/4.3. The topic name is based on the path with removed leading and ending slashes.

Make sure to replace the MQTT_FIRMWARE_URL value with your MQTT server credentials, host and topic.

Security
--------
For additional security a standard SSL/TLS can be used
Expand Down
1 change: 1 addition & 0 deletions Sming/Libraries/OtaUpgradeMqtt/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ OTA Upgrade over MQTT classes

.. doxygenclass:: OtaUpgrade::Mqtt::PayloadParser
.. doxygenclass:: OtaUpgrade::Mqtt::RbootPayloadParser
.. doxygenclass:: OtaUpgrade::Mqtt::AdvancedPayloadParser
32 changes: 31 additions & 1 deletion Sming/Libraries/OtaUpgradeMqtt/component.mk
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,34 @@ COMPONENT_VARS += ENABLE_OTA_VARINT_VERSION
ENABLE_OTA_VARINT_VERSION ?= 1

COMPONENT_CXXFLAGS := -DENABLE_OTA_ADVANCED=$(ENABLE_OTA_ADVANCED) \
-DENABLE_OTA_VARINT_VERSION=$(ENABLE_OTA_VARINT_VERSION)
-DENABLE_OTA_VARINT_VERSION=$(ENABLE_OTA_VARINT_VERSION)

##@Firmware Upgrade

OTA_TOOLS := $(COMPONENT_PATH)/tools
OTA_DEPLOYMENT_TOOL = $(OTA_TOOLS)/deployer/out/Host/debug/firmware/deployer$(TOOL_EXT)

$(OTA_DEPLOYMENT_TOOL):
$(Q) $(MAKE) -C $(OTA_TOOLS)/deployer SMING_ARCH=Host ENABLE_CUSTOM_LWIP=2


# SDP = Sming Deployment Pakage
OTA_PACKAGE_EXT =.sdp

OTA_PATCH_VERSION ?= $(shell date +%s)

PACKAGE_IN = $(RBOOT_ROM_0_BIN)
ifneq ($(ENABLE_OTA_ADVANCED), 0)
PACKAGE_IN = $(OTA_UPGRADE_FILE)
endif
PACKAGE_OUT = $(PACKAGE_IN)$(OTA_PACKAGE_EXT)

.PHONY: ota-pack
ota-pack: $(PACKAGE_OUT) ##Creates a deployment package from the current application (use OTA_PATCH_VERSION to specify the version number)

$(PACKAGE_OUT): $(PACKAGE_IN) $(OTA_DEPLOYMENT_TOOL)
$(Q) $(OTA_DEPLOYMENT_TOOL) pack -- $(PACKAGE_IN) $(PACKAGE_OUT) $(OTA_PATCH_VERSION) $(ENABLE_OTA_VARINT_VERSION) 2>/dev/null

.PHONY: ota-deploy
ota-deploy: $(PACKAGE_OUT) ##Uploads new firmware version of the current application (use MQTT_FIRMWARE_URL to specify the MQTT URL)
$(Q) $(OTA_DEPLOYMENT_TOOL) deploy -- $(PACKAGE_OUT) $(MQTT_FIRMWARE_URL) 2>/dev/null
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ void otaUpdate()
mqtt.setPayloadParser([parser](MqttPayloadParserState& state, mqtt_message_t* message, const char* buffer,
int length) -> int { return parser->parse(state, message, buffer, length); });

String updateTopic = "/a/";
String updateTopic = "a/";
updateTopic += APP_ID;
updateTopic += "/u/";
updateTopic += APP_VERSION;
Expand Down
Empty file.
97 changes: 97 additions & 0 deletions Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.cproject
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?fileVersion 4.0.0?><cproject storage_type_id="org.eclipse.cdt.core.XmlProjectDescriptionStorage">
<storageModule moduleId="org.eclipse.cdt.core.settings">
<cconfiguration id="0.131700304">
<storageModule buildSystemId="org.eclipse.cdt.managedbuilder.core.configurationDataProvider" id="0.131700304" moduleId="org.eclipse.cdt.core.settings" name="Default">
<externalSettings/>
<extensions>
<extension id="org.eclipse.cdt.core.VCErrorParser" point="org.eclipse.cdt.core.ErrorParser"/>
<extension id="org.eclipse.cdt.core.GmakeErrorParser" point="org.eclipse.cdt.core.ErrorParser"/>
<extension id="org.eclipse.cdt.core.CWDLocator" point="org.eclipse.cdt.core.ErrorParser"/>
<extension id="org.eclipse.cdt.core.GCCErrorParser" point="org.eclipse.cdt.core.ErrorParser"/>
<extension id="org.eclipse.cdt.core.GASErrorParser" point="org.eclipse.cdt.core.ErrorParser"/>
<extension id="org.eclipse.cdt.core.GLDErrorParser" point="org.eclipse.cdt.core.ErrorParser"/>
</extensions>
</storageModule>
<storageModule moduleId="cdtBuildSystem" version="4.0.0">
<configuration buildProperties="" description="" id="0.131700304" name="Default" optionalBuildProperties="" parent="org.eclipse.cdt.build.core.prefbase.cfg">
<folderInfo id="0.131700304." name="/" resourcePath="">
<toolChain id="org.eclipse.cdt.build.core.prefbase.toolchain.481709424" name="No ToolChain" resourceTypeBasedDiscovery="false" superClass="org.eclipse.cdt.build.core.prefbase.toolchain">
<targetPlatform id="org.eclipse.cdt.build.core.prefbase.toolchain.481709424.1637922483" name=""/>
<builder id="org.eclipse.cdt.build.core.settings.default.builder.1861624051" keepEnvironmentInBuildfile="false" managedBuildOn="false" name="Gnu Make Builder" superClass="org.eclipse.cdt.build.core.settings.default.builder"/>
<tool id="org.eclipse.cdt.build.core.settings.holder.libs.927548352" name="holder for library settings" superClass="org.eclipse.cdt.build.core.settings.holder.libs"/>
<tool id="org.eclipse.cdt.build.core.settings.holder.1318259878" name="Assembly" superClass="org.eclipse.cdt.build.core.settings.holder">
<inputType id="org.eclipse.cdt.build.core.settings.holder.inType.49095807" languageId="org.eclipse.cdt.core.assembly" languageName="Assembly" sourceContentType="org.eclipse.cdt.core.asmSource" superClass="org.eclipse.cdt.build.core.settings.holder.inType"/>
</tool>
<tool id="org.eclipse.cdt.build.core.settings.holder.451866905" name="GNU C++" superClass="org.eclipse.cdt.build.core.settings.holder">
<inputType id="org.eclipse.cdt.build.core.settings.holder.inType.952230018" languageId="org.eclipse.cdt.core.g++" languageName="GNU C++" sourceContentType="org.eclipse.cdt.core.cxxSource,org.eclipse.cdt.core.cxxHeader" superClass="org.eclipse.cdt.build.core.settings.holder.inType"/>
</tool>
<tool id="org.eclipse.cdt.build.core.settings.holder.639761705" name="GNU C" superClass="org.eclipse.cdt.build.core.settings.holder">
<inputType id="org.eclipse.cdt.build.core.settings.holder.inType.2127747414" languageId="org.eclipse.cdt.core.gcc" languageName="GNU C" sourceContentType="org.eclipse.cdt.core.cSource,org.eclipse.cdt.core.cHeader" superClass="org.eclipse.cdt.build.core.settings.holder.inType"/>
</tool>
</toolChain>
</folderInfo>
</configuration>
</storageModule>
<storageModule moduleId="org.eclipse.cdt.core.externalSettings">
<externalSettings containerId="Sming;" factoryId="org.eclipse.cdt.core.cfg.export.settings.sipplier"/>
</storageModule>
</cconfiguration>
</storageModule>
<storageModule moduleId="cdtBuildSystem" version="4.0.0">
<project id="Basic_ControlPoint.null.1347473968" name="Basic_ControlPoint"/>
</storageModule>
<storageModule moduleId="scannerConfiguration">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
<scannerConfigBuildInfo instanceId="0.2058741737">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
<scannerConfigBuildInfo instanceId="cdt.managedbuild.toolchain.gnu.mingw.base.1135534147;cdt.managedbuild.toolchain.gnu.mingw.base.1135534147.86962463;cdt.managedbuild.tool.gnu.c.compiler.mingw.base.2032390008;cdt.managedbuild.tool.gnu.c.compiler.input.1700912488">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
<scannerConfigBuildInfo instanceId="0.147001341">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
<scannerConfigBuildInfo instanceId="0.1962726862">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
<scannerConfigBuildInfo instanceId="0.358052692">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
<scannerConfigBuildInfo instanceId="0.131700304">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
<scannerConfigBuildInfo instanceId="cdt.managedbuild.toolchain.gnu.mingw.base.1135534147;cdt.managedbuild.toolchain.gnu.mingw.base.1135534147.86962463;cdt.managedbuild.tool.gnu.cpp.compiler.mingw.base.454898447;cdt.managedbuild.tool.gnu.cpp.compiler.input.501261625">
<autodiscovery enabled="true" problemReportingEnabled="true" selectedProfileId=""/>
</scannerConfigBuildInfo>
</storageModule>
<storageModule moduleId="org.eclipse.cdt.core.LanguageSettingsProviders"/>
<storageModule moduleId="org.eclipse.cdt.make.core.buildtargets">
<buildTargets>
<target name="all" path="" targetID="org.eclipse.cdt.build.MakeTargetBuilder">
<buildCommand>make</buildCommand>
<buildArguments/>
<buildTarget>all</buildTarget>
<stopOnError>true</stopOnError>
<useDefaultCommand>true</useDefaultCommand>
<runAllBuilders>true</runAllBuilders>
</target>
<target name="rebuild" path="" targetID="org.eclipse.cdt.build.MakeTargetBuilder">
<buildCommand>make</buildCommand>
<buildArguments/>
<buildTarget>rebuild</buildTarget>
<stopOnError>true</stopOnError>
<useDefaultCommand>true</useDefaultCommand>
<runAllBuilders>true</runAllBuilders>
</target>
<target name="flash" path="" targetID="org.eclipse.cdt.build.MakeTargetBuilder">
<buildCommand>make</buildCommand>
<buildArguments/>
<buildTarget>flash</buildTarget>
<stopOnError>true</stopOnError>
<useDefaultCommand>true</useDefaultCommand>
<runAllBuilders>true</runAllBuilders>
</target>
</buildTargets>
</storageModule>
</cproject>
30 changes: 30 additions & 0 deletions Sming/Libraries/OtaUpgradeMqtt/tools/deployer/.project
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>FirmwareDeployer</name>
<comment></comment>
<projects>
<project>SmingFramework</project>
<project>Libraries</project>
<project>Sming</project>
</projects>
<buildSpec>
<buildCommand>
<name>org.eclipse.cdt.managedbuilder.core.genmakebuilder</name>
<triggers>clean,full,incremental,</triggers>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>org.eclipse.cdt.managedbuilder.core.ScannerConfigBuilder</name>
<triggers>full,incremental,</triggers>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>org.eclipse.cdt.core.cnature</nature>
<nature>org.eclipse.cdt.core.ccnature</nature>
<nature>org.eclipse.cdt.managedbuilder.core.managedBuildNature</nature>
<nature>org.eclipse.cdt.managedbuilder.core.ScannerConfigNature</nature>
</natures>
</projectDescription>
9 changes: 9 additions & 0 deletions Sming/Libraries/OtaUpgradeMqtt/tools/deployer/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#####################################################################
#### Please don't change this file. Use component.mk instead ####
#####################################################################

ifndef SMING_HOME
$(error SMING_HOME is not set: please configure it as an environment variable)
endif

include $(SMING_HOME)/project.mk
62 changes: 62 additions & 0 deletions Sming/Libraries/OtaUpgradeMqtt/tools/deployer/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
Device_Scanner
==============

.. highlight:: bash

Utility to package and deploy firmware files for upgrade over MQTT
See :library:`OtaUpgradeMqtt` for details.


Packaging
---------

To scan your local network, do this::

make pack HOST_PARAMETERS=scan


This will start by scanning for all root devices ``:upnp:rootdevice``.
If you want to use a specific search type, just add it to the command line::

make run HOST_PARAMETERS='scan urn:schemas-upnp-org:service:ContentDirectory:1'

This will take some time to complete. If there are any failures, you can retry
by running the command again: any descriptions which have already been fetched will
be skipped.

You find output in the following directories:

out/upnp/devices
Contains a hierarchical map of all found devices on your local UPnP network.


The schema directories are populated with modified versions of the description files.
These reflect the C++ class structures which we will later create:

out/upnp/schema/services
All services are extracted into a directory by ``{domain}``. Each service has a separate .xml file.

Service schema are re-usable and new ones should be added to the UPnP library so they are available for
everyone to use.


out/upnp/schema/devices
All devices (both root and embedded) are extracted to a separate directory ``{manufacturer}/{friendlyName}``
with a single .xml file for each device.

Device schema are specific to each device implementation so will generally be kept in a separate library
or application.


Processing existing descriptions
--------------------------------

You can also process device descriptions files pulled in via other means. For example:

make run HOST_PARAMETERS='parse config/panasonic/viera dmr/ddd.xml dms/ddd.xml nrc/ddd.xml'

The first parameter ``parse`` is the command.
The second parameter ``config/panasonic/viera`` gives the root directory for the device.
The remaining parameters are the relative locations from this directory of a device description file.
References to service files are pulled in: if they are missing, this may fail.

Loading

0 comments on commit 0a9b28a

Please sign in to comment.