In this article, I aim to debunk how to approach the testing of robotics software. Robotics involves intricate microprocessors, embedded systems, and diverse software stacks. Software quality engineers play a crucial role in testing these intricate software components to ensure seamless functionality and performance. I will explore the different software stacks utilized in robotics and shed light on the essential testing methodologies used by software quality engineers in this dynamic and evolving field.
Robotics Software Design
In the world of robotics, the key items in software design are hardware, microprocessors, embedded systems, sensors, and software. The software piece is heavily reliant on the performances of all the above components. To elaborate more on the software design, it can comprise so many things starting from the low-level platform and kernel design to middleware layers to the more user-facing app layers. Understanding these layers is super critical to write a test automation design for these components.
Platform Layer
This is where the core platform components are built. This is the lowest level of component, which is built on top of ROS (Robot OS). ROS commands are used widely to issue a move command to the robot and increase or decrease speed and mobility. This layer is supercritical, and all the top-level components end up calling this layer for the actual motion of the device.
Mobility Layer
This is the layer that interacts between the platform and middleware layer. This provides a series of restful apps, which can be called by various clients on the top level. They communicate the command to the platform layer for translation into motion command.
Middleware Layer
This is the intermediary layer which acts as a bridge between the upper client and the low platform layer. All the front-end apps talks to this layer through an internal IP connection, and this layer constantly monitors the various states of the robot. Examples are idle, motion, turning, etc.
Front-End/User-Facing Layer
These are front-end user-facing apps. Consider a smart home robot, the user-facing apps could be a homepage app, calendar app, or default apps to control the device settings. Any integrations with Voice assistants like OK Google or Amazon Alexa.
How To Approach Testing This Software Stack
Testing should be done at all individual layers, and we need to do a ton of E2E (End-to-End) testing where all three layers are tested together.
The second category of testing is testing the performance of the product. This is also crucial since any consumer robots with poor performances are not ideal. Having said all of this, I will help you break down the pieces on how to start approaching software test automation.
Functional Test Automation Approach
Here we will see how to approach test automation for validating the functionality. As I mentioned earlier, testing can and should be done at various layers listed above. The overall functional testing can be split into two major sections.
- Component Testing
- End-to-End Testing
Component Testing
Below are some of the examples of component testing use cases. These tests are aimed at covering the depth of testing of each component. Some component test scenarios include and are not limited to,
Platform Layer:
- Testing of safety features. Robots should not cause damage to users. They need to adhere to safety conditions.
- Thermal temperature tests. How are the temperature values increase when a device is in use?
- Sensor readings, values, image processing, etc.
Mobility Layer:
- Capturing mobility speed, acceleration, and turning radius.
- Navigation in a pre-defined or unknown region.
Middleware Layer:
- Connection between the low-level and higher-level components.
- Logging, error handling, and reporting.
Front End App Layer:
- Covering various scenarios on how a user can interact with the robot
End to End Testing, a.k.a E2E Testing
There is a myth where often developers conclude that components are tested to maximum depth, and that is why we need to invest time in E2E testing. The truth is we are not shipping the components separately to the end customers. It is the entire Robot that is shipped where all the components are integrated. Testing at this layer is very crucial and cannot be skipped — for example, tests for home vacuum cleaner robots.
- Can the robot navigate from the living room to the kitchen
- Can the robot go back to the charger?
- Does the robot hit my cat while moving?
These are some of the samples, but this is the toughest piece to automate and test them under a closed and clean test environment. This is very critical to identify software bugs in the process.
Performance Test Automation Approach
Shipping a product with a ton of cool features and poor performance is useless. Hence focusing on testing the software and hardware components at the Performance layer is important.
Performance tests are targeted to get the latency and reliability of the software. Latency is the time taken for an action to be performed and represented by percentile. Tp90 for the Top speed of the robot is 0.2 m/s. Tp95 for responding to mobility commands is 500ms. These are measured against some standard KPIs (Key Performance Indices) to measure the performance quality of the product.
Reliability is very straightforward, as the name suggests how is my software reliable over some time
What’s Next?
Hope you would have got a basic understanding of the SW design of a robot and a different approach to testing. In my upcoming articles, I will dive deeper into various test automation techniques with some real examples.
