How do you test the performance of Internet of Things (IoT) SIMs in the field? Our new white paper covers this topic in depth with real results from the field.
In this article and accompanying white paper, Asia Mobiliti takes you through their testing process for comparing SIM performance and real-world results.
Read on to find out how to compare SIMs based on the best signal quality, power consumption, and data consumption, or download the full report right now.
Who is Asia Mobiliti?
Asia Mobiliti makes connected data products and powers mobility solutions that move people and things in a way that is smart, seamless, and good for the environment.
They believe that digitizing mobility is a key part of fixing problems like traffic, pollution, and underused transit services in emerging markets.
Why test connectivity?
Connecting IoT devices to cellular networks requires SIM cards. They are the key to a stable and secure connection to cellular networks and the cloud.
Cellular connectivity is a surprisingly varied field, however. There are big differences between 2G, 3G, 4G, 5G, LTE-M, and NB-IoT, and there are also many operators who connect to cellular networks with varying qualities of service and feature availability.
To make sure you have the right connectivity for your device—connectivity that is scalable and ready for the future—it’s important to test connectivity performance and compare operators.
If you don’t test your IoT devices correctly, they will become an expensive paperweight. That’s the beautifully concise way that Join Howie Lim U, Director of Technology at Asia Mobiliti, put it in a recent webinar hosted by Onomondo.
As good as IoT is for building a digital twin of the assets that you’re monitoring, it can equally become the greatest problem you have to face if testing isn’t conducted properly.
How do you conduct SIM performance tests?
First, you’ll want to define the deployment parameters and conditions you would like to test devices in.
Ask questions like:
- Will the devices be stationary or mobile?
- Will they be in an urban or rural setting or both?
- Do they need to react quickly to messages or can they be powered down for long intervals?
- Do the devices have a reliable external power source available or are they battery powered?
Then, it’s time to define the test profile. This comes once you’ve defined your deployment requirements. In the Asia Mobiliti use case, the test prioritization, in descending order, was:
- To reduce data consumption
- Maximize reliability and availability of connection
- Guarantee data security
- Optimize power consumption
Now that Asia Mobiliti’s test priorities are clear, let’s look at the test results.
SIM performance tests conducted in the report
Asia Mobiliti tested four operators in Malaysia, a set that included MNVOs and MNOs. The tests happened concurrently, all with the same device configuration.
The first round of tests was a network signal and coverage test. One major point of interest was evaluating the impact of PLMN lists on connection stability.
Unlike most MVNOs and MNOs, Onomondo SIMs do not have PLMN lists by default; they are optional to install on SIMs. By letting SIMs choose networks on their own, coverage and device performance can be improved by making sure connections are made based on quality and not on a list.
Additionally, Asia Mobiliti tested Onomondo’s TLS Connectors, which move cloud SDKs into the network, allowing Asia Mobiliti to abstract data security, encryption, and cloud provisioning from devices. Onomondo’s Connectors give businesses the ability to easily control endpoints remotely, while also reducing data packet size and power consumption.
Signal quality tests
Gauging network availability and reliability is key for Asia Mobiliti, as their devices are always moving about. Additionally, field tests were a high priority, as you don’t get random variables like disconnections and poor coverage in labs.
In Asia Mobiliti’s network roaming tests, Onomondo had the highest average signal quality.
| Round 1 Average signal quality | Round 2 Average signal quality | Total average | |
|---|---|---|---|
| Onomondo | 88.61 % | 91.56 % | 90.09 % |
| Operator A | 86.70 % | 90.33 % | 88.52 % |
| Operator B | 67.66 % | 85.00 % | 76.33 % |
| Operator C | 90.43 % | 85.11 % | 87.77 % |
Onomondo also had the equal least number of disconnecting events. Interestingly, Operator A stands out with a relatively high number of disconnection events. This could be attributed to their use of a PLMN list, something that highlights the need for testing SIMs with and without PLMNs.
| Round 1 Disconnection events | Round 2 Disconnection events | Total average | |
|---|---|---|---|
| Onomondo | 2 | 0 | 1 |
| Operator A | 5 | 4 | 4.5 |
| Operator B | 1 | 2 | 1.5 |
| Operator C | 1 | 1 | 1 |
Data efficiency and power consumption tests
Optimizing data efficiency was priority number one for Asia Mobiliti.
By reducing data transmission, you can
- reduce power consumption as devices spend less time actively transmitting data,
- you reduce the chances of packet loss issues,
- and you spend less money on data.
When testing Onomondo’s TLS Connector in an ideal lab test setup, Asia Mobiliti saw upwards of 75% data savings for messages sent every five minutes.
| Lab Test Condition | Data usage (KB) |
|---|---|
| With TLS Connector | 1,200 |
| Bypass (w/o Connector) | 4,800 |
In a real-world scenario with a messaging interval of 10 seconds, Asia Mobiliti saw data savings of over 33% with connectors. The lower savings can be put down to the shorter message intervals and varying network quality, including disconnects and DNS changes that happen in the field.
| Field Test Condition | Data usage (KB) |
|---|---|
| With TLS Connector | 60,455 |
| Bypass (w/o Connector) | 90,574 |
Even though Asia Mobiliti’s test framework gave improved power efficiency the lowest priority, the devices still have internal batteries that would benefit greatly from being more efficient.
In Asia Mobiliti’s power consumption tests, they saw savings of 12.79% when using Onomondo Connectors. This may not seem like a lot at first, but when you extrapolate the extra battery life across thousands of devices, you see some real gains.
Read the full report Asia Mobiliti report
To get the full picture from Asia Mobiliti’s SIM performance tests, download the full report available now via the form below.
