Debugging

Debugging Stacktraces

When Fruitymesh reboots, it will try to save the reason in the ramRetainStruct. The Stacked Program Counter (PC before e.g. the hardfaultHandler was called) will be saved in stacktrace[0]. Using this information we should be able to get the source code line where the error happened. Use the following:

<gccDir>\bin\arm-none-eabi-addr2line -e <fruitymeshDir>\_build\debug\NRF51_BOARD\FruityMesh.out 0x35BA5

The address must be given as a hex value, otherwise, addr2line will print no valid result!! FruityMesh must have been compiled with debug information using -g

Reading the flash of a broken FruityMesh Node

In order to analyze a broken node, the flash and UICR can be read back in order to analyze the problem. A good idea is to read back UICR and flash and afterwards compare the .hex with either the last .hex flashed onto this beacon (from the fruitydeploy beacon directory) or flash the beacon again and read it back.

Read back code and UICR to a .hex file:

nrfjprog --readuicr --readcode E:\dump.hex

Convert both .hex files to a hex dump so we can view them with a standard txt editor:

srec_cat C:\test\dump.hex -intel -o C:\test\dump.txt -hex_dump

Afterwards, use The TortoiseDiff Tool to see the differences:

  • Page 0 is the master boot record that should not have been altered

  • The first ~120kb are the softdevice that should not have been altered

  • Afterwards comes the application that should not have changes

  • The free space after the application can be whatever it wants

  • The settings are located 2 pages before the bootloader:

    • 0x3E400 (NRF51)

    • 0x76000 (NRF52)

  • The bootloader is at:

    • #define FRUITYLOAD_ADDRESS 0x3EC00 (NRF51)

    • #define FRUITYLOAD_ADDRESS 0x78000 (NRF52)

In order to check the correctness of the settings, we could convert the hex dump of the settings pages to a c array using sublime and load it into the flash of a node in the simulator. Then we can properly debug the startup behaviour. But debugging a real node should also be possible if only the settings are broken.

Debug Running node

When the nodes are somewhere in the building and one of them is stuck in a hardfault, or another endless loop, debugging is complicated. Here’s how to do it:

  • If the node is connected with its USB cable, put a battery in the node, then unplug the USB cable

  • Take the node to the workstation, while it is running on battery

  • Use two USB cables that are both plugged into the docking station (do not plug one into the laptop and the other in the docking station because they might have a different ground!!!)

  • One of the cables is plugged into the DevKit / Debugger

  • The other cable is plugged into the node ⇒ Debugger and Node are now powered from the same ground level, this is important

  • Next, connect the Debug cable to the node

  • In Eclipse, select the Debug Configuration "Running Target". (Connect to running target must be selected and no reset, etc…​ shall be performed)

  • The node should not have restarted during this procedure, …​ hopefully. * == Finding problems in a running System

Often, an error only occurs seldomly and in a system that is only remotely accessible. FruityMesh has some inbuilt functionality to find errors in these cases. First, there are live reports (see StatusReporterModule). You could add a number of live report statements to the code and they will be reported through the mesh, as soon as they trigger. The only problem with live reports is, that you might lose them if they always occur if a mesh connection is lost for example. In this case, you can use the error logging facility of the Logger class. This way, the error is logged to ram and can be requested once the node is in the mesh again.

nRF Sniffer

The nRF sniffer is used with wireshark. A custom dissector for fruitymesh JOIN_ME packets has been written and is saved in the util/wireshark folder. Integration into wireshark is easy and documented at the beginning of the lua file.

The nRF Sniffer does not work with an updated SEGGER Firmware, it is necessary to put the SEGGER Firmeware JLink_V498c on the beacon by using JLinkConfig.exe and replacing selecting replace firmware. Afterwards, the sniffer should work together with the fixed sniffer application received by nordic.

If the fixed sniffer application does not find the beacon, open the buggy official version first until the beacon is found, then close and reopen the fixed version.