International Research Journal of Engineering Science, Technology and Innovation

Analyzing Data Usage and Cost Considerations among Cloud Service Providers for IoT Sensor Data Processing

Anton Shykhmat^* and Zenoviy Veres Department of Computerized Automatic Systems, Lviv Polytechnic National University, Ukraine
^*Corresponding Author:
Anton Shykhmat, Department of Computerized Automatic Systems, Lviv Polytechnic National University, Ukraine, Email: anton.o.shykhmat@lpnu.ua

Received: 16-Sep-2023, Manuscript No. irjesti-24-114040; Editor assigned: 19-Sep-2023, Pre QC No. irjesti-24-114040 (PQ); Reviewed: 03-Oct-2023, QC No. irjesti-24-114040; Revised: 10-Jan-2025, Manuscript No. irjesti-24-114040 (R); Published: 17-Jan-2025, DOI: 10.14303/2315-5663.2025.118

Introduction

Internet of Things (IoT) is widely adopted within every aspect of our life. IoT enables the creation of networks between devices, people, and applications on the Internet, resulting in ecosystems with higher productivity, better energy efficiency, and greater profitability. Devices help to recognize the state of affairs, which gives them the advantage of anticipating a person's needs based on information gathered by context (Hanes D et al., 2017). COVID-19 increased the remote work demand. It raises tasks to collect, process, store and figure out the insights from the received data. Being able to manage a massive amount of devices within the system is a complex task by itself (Pierleoni P et al., 2019). An increased number of devices adds extra price to build a robust solution to receive the telemetry data, check their state, and discover disconnected/failed ones proactively. Cloud is commonly considered as a basis to build the solution for the IoT field (Misra S et al., 2021). The most straightforward (and most challenging at the same time) approach is to use cloud computation capabilities and set up all required components on your own. However, AWS and GCP cloud provides provide cloud IoT core modules to set up, manage, and ingest telemetry to the cloud. Both IoT Core solution supports data ingest using two widely adopted protocols in the IoT field: MQTT and HTTP (Maurya R et al., 2021).

The rest of the paper is organized as follows. Section II provides an overview of the following IoT data protocols: MQTT and HTTP. Section III compares amount of billed traffic produced by each protocol and expenses associated with this traffic. Section IV concludes the paper by providing decision tree to select the best-fit protocol for particular use cases (Atmoko RA, et al., 2017).

Literature Review

Protocols overview

MQTT: MQTT is a messaging protocol for the Internet of Things (IoT) developed and managed by the OASIS MQTT technical committee. It is lightweight, open, simple, and designed to be easy to implement. These characteristics make it ideal for use in many situations, including constrained environments such as for communication in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a small code footprint is required and/or network bandwidth is at a premium (Amadeo M, et al., 2015).

MQTT provides the ability to have the device connected indefinitely even if it does not transmit any data. The broker keeps track of connected devices using the keepalive feature. The Keep-Alive is a time interval measured in seconds. It is expressed as a 16-bit word; it is the maximum time interval permitted to elapse between the point at which the client finishes transmitting one control packet and the point it starts sending the next. It is the responsibility of the client to ensure that the interval between control packets being sent does not exceed the keep alive value. In the absence of sending any other control packets, the client MUST send a PINGREQ packet (Longo E, et al., 2020).

When a connection is lost, the broker could issue the client’s Last Will and Testament (LWT) message. The message could be used as a trigger to notify the user about the issue and proactively figure out the disconnect reason. However, disconnect could occur due to connectivity issues and could be a false positive. As a consequence, a more robust approach is required to figure out the device’s failure (Ali AA, 2018).

The MQTT specification describes three Quality of Service (QoS) levels:

• QoS 0, delivered at most once
• QoS 1, delivered at least once
• QoS 2, delivered exactly once

Please note, Cloud IoT Core does not support QoS 2. Publishing QoS 2 messages close the connection.

According to the GCP documentation, Cloud IoT Core limits the max inactivity period with idle time set to 20 minutes: "A client connection will automatically be terminated if the client doesn't send any messages for 20 minutes, even if the keep-alive interval is longer. If a keep-alive value isn't supplied, the default idle timeout of 20 minutes still takes effect".

HTTP: HTTP was invented as a World Wide Web component to transfer documents. It is most familiar to us as an enabling technology that allows web browsers to work. Servers contain resources identified by the URLs to which HTTP clients can usually make requests. HTTP is a "connectionless" protocol: devices do not maintain a connection to Cloud IoT Core with the HTTP bridge. Instead, they send requests and receive responses. Cloud IoT Core supports HTTP 1.1 only (not 2.0). HTTP bridge could be used to send the device state to the IoT Core regularly.

REST is an architectural style for building web services based on the HTTP protocol. Services that support this style are called RESTful services. Such services do not store the client's state, making their usage fast, reliable, and scalable. In response to requests made to a resource URI, RESTful services often respond in HTML, JSON, or XML formats (but are not limited to these). RESTful services most often use the following 4 HTTP methods:

• GET: To retrieve resource information only and do not modify it.
• POST: To create new resources.
• PUT: To update existing resources.
• DELETE: To delete a current resource.

Discussion

Billed traffic comparison

According to the tests performed to deliver 1 K messages over MQTT and HTTP - MQTT was shown 6 times faster on the task of posting consistent time-valuable data and is more efficient from a power consumption point of view.

Set of scenarios were evaluated to compare the billed traffic by GCP and AWS that represents the most common patterns for data transmission in the IoT field:

• 1 Kb message payload, data is transmitted every minute
• 1 Kb message payload, data is transmitted every 5 mins
• 1 Kb message payload, data is transmitted every 10 mins
• 1 Kb message payload, data is transmitted every 15 mins
• 1 Kb message payload, data is transmitted every 20 mins
• 1 Kb message payload, data is transmitted every 30 mins
• 1 Kb message payload, data is transmitted every hour
• 1 Kb message payload, data is transmitted every 2 hours
• 1 Kb message payload, data is transmitted every 3 hours
• 1 Kb message payload, data is transmitted every 6 hours

For the MQTT bridge, PINGREQ or data message should be delivered at least once per 20 minutes to maintain the connection opened. In GCP, PINGREQ are charged in the same way, as data messages. AWS does not charge for PINGREQ messages, but it charges for total connection time. The GCP minimum billed message is 1 Kb, even if the message itself is only a few bytes, for AWS minimum billed message is 5 Kb. For HTTP bridge, both GCP and AWS bill every request and response with data transmission. The minimum billed message size is also 1 Kb for GCP and 5 Kb for AWS. Let’s compare HTTP vs. MQTT bridges with PINGREQ message is transmitted from each device every 20 mins for 10 k, 100 k, 1 M, 10 M, and 100 M devices. The calculations assume that each device connects/re-connects to the MQTT Bridge only once per day (Connection is billed as 1 Kb message by GCP, for AWS it is billed as the size of the message, so let’s assume it also 1 Kb). More detailed calculations are presented below for scenarios: 1, 5, 6 and 7.

Scenario 1: 1 Kb message payload, data is transmitted every minute

The data transmission frequency adds a crucial amount of traffic. PINGREQ messages are approximately 5% of all traffic and are relatively small, and their contribution could be ignored for calculations. HTTP bridge is used almost 2x more messages since it is billed for each request and response separately. As a consequence, the HTTP bridge is not applicable for the scenario with high message transmission frequency (Table 1).

Table 1. Billed traffic for MQTT and HTTP bridges for scenario 1 with PINGREQ message every 20 minutes.

Scenario 5: 1 Kb message payload, data is transmitted every 20 minutes

As it is presented in Table 2, HTTP Bridge billed traffic is comparable to MQTT in GCP (for AWS MQTT billed traffic is still twice smaller).

The difference between bridges is caused by the assumption of MQTT connection/reconnection frequency.

This assumption adds approximately 0.7% of extra traffic. The main option to decrease the billed traffic cost is to extend the PINGREQ message time from 20 minutes up to the maximum possible value for MQTT according to the specification - 18 hours. It will decrease the PINGREQ traffic 54 times per device, as stated in Table 3.

Table 2. Billed traffic for MQTT and HTTP bridges for scenario 5 with PINGREQ message every 20 minutes.

Table 3. Billed traffic for MQTT and HTTP bridges for scenario 5 with PINGREQ message every 18 hours.

Increasing the PINGREQ time saves almost 49% of billed traffic for the MQTT bridge in GCP and has no changes for AWS. Since each HTTP transmission is billed as two 1 Kb messages compared to 1 Kb messages for MQTT, the MQTT bridge is a cheaper and preferred approach to transmit the data for scenarios with frequent data transmissions (less than 20 minutes) and stable network connections.

The connection’s traffic shows that MQTT is preferable over HTTP bridge even the reconnect needs to happen every 20 minutes the connect traffic will be equal to the telemetry messages traffic.

Scenario 6: 1 Kb message payload, data is transmitted every 30 minutes

This scenario is evaluated with the following assumptions:

• PINGREQ message is delivered every 20 minutes for the MQTT bridge.
• The connection/re-connection is made only once per day for the MQTT bridge (Table 4).

Table 4. Billed traffic for MQTT and HTTP bridges for scenario 6.

Thirty minutes telemetry messages interval is when the GCP MQTT bridge requires 26% more billed traffic than the HTTP ones, but AWS MQTT bridge requires 48.95% less billed traffic than the HTTP.

Scenario 7: 1 Kb message payload, data is transmitted every hour

This scenario is evaluated with the following assumptions:

• PINGREQ message is delivered every 20 minutes for the MQTT bridge.
• The connection/re-connection is made only once per day for the MQTT bridge.

As presented in Table 5, MQTT PINGREQ messages add more than 74% of billed traffic to the GCP MQTT bridge. This contribution increases when the message delivery is done less frequently: PINGREQ messages add 83% of billed traffic if telemetry message is delivered every second hour, and it adds 90% of traffic for scenario 10 (telemetry message is delivered every 6 hours).

Table 5. Billed traffic for MQTT and HTTP bridges for scenario 7.

Expenses comparison

The billed traffic size is not the only criterion to select the solution for connecting IoT devices to the cloud. Expenses are also a crucial point for a business that constraints architecture design. As mentioned earlier, AWS does not charge for PINGREQ messages, but it charges for device connection time. Table 6 represents expenses calculation for different transmission scenarios using GCP MQTT bridge with PINGREQ frequency 20 mins, Table 7 represents expenses calculation for AWS MQTT bridge when device is connected to MQTT bridge for the whole day, Table 8 represents expenses calculation for GCP HTTP bridge, and Table 9 represents expenses calculation for AWS HTTP bridge (scenarios 6-10).

Table 6. Traffic expenses for GCP MQTT bridge, 1 Kb message payload.

Table 7. Traffic expenses for AWS MQTT bridge, 1 Kb message payload.

Table 8. Traffic expenses for GCP HTTP bridge, 1 Kb message payload.

Table 9. Traffic expenses for AWS HTTP bridge, 1 Kb message payload.

The HTTP bridge could be used for data transmission for 10 k up to 1 M devices when it happens every 30 minutes or less often. The price comparison shows a massive increase between 1 M and 10 M devices in GCP. While the device count increased by 10, the GCP MQTT expenses increased 18x for 6 hours of data transmission and almost 20x for 30 mins transmission frequency. The HTTP bridge price is also growing 10x between 1 M and 10 M devices. For AWS situation is different. Price per message decrease when you produce more traffic, and as we can see HTTP protocol is cheaper, even it produces more billable traffic. Also, we may notice, that AWS MQTT bridge is 3x cheaper than GCP MQTT bridge, and AWS HTTP bridge is 6x cheaper than GCP HTTP bridge (Table 10).

As mentioned in the scenario 5 section, the GCP MQTT could be configured to deliver a PINGREQ message every 18 hours to save traffic. The expenses for such conditions (scenarios 1-5) are represented below:

Table 10. Traffic expenses for GCP MQTT bridge, 1 Kb message payload with PINGREQ message every 18 hours.

As of AWS, we can decrease charges, if we do not need to connect device to MQTT bridge for the whole day, but only for specific working hours. The expenses for 8 connection hours per day (scenarios 1-5) are represented in Table 11 (for MQTT) and in Table 12 (for HTTP).

Table 11. Traffic expenses for AWS MQTT bridge, 1 Kb message payload, 8 hours per day.

Table 12. Traffic expenses for AWS HTTP bridge, 1 Kb message payload, 8 hours per day.

Overall AWS IoT Core is 5 times cheaper than GCP IoT Core for both MQTT and HTTP protocols, and MQTT bridge in AWS is about 45% cheaper than HTTP bridge in AWS for frequent data transmission scenarios (<5 mins) (Table 13). The decision table with recommended options for telemetry delivery as a summary of the research is depicted below:

Table 13. Protocols selection decision tree.

Conclusion

The data transmission depends on the business requirements. Set of use cases and more detailed approach for bridge selection is described above. The major driver for the proper selection the telemetry delivery frequency and devices count. As mentioned earlier, GCP IoT Core is more expensive comparing to AWS IoT Core for all evaluated scenarios, so it is not recommended to use. For almost all cases usage of AWS IoT Core MQTT bridge is applicable for frequent data delivery. The device count increase (>10 M) with high frequency of data delivery (approx. every 1 minute) push the solution to use either standalone MQTT broker or figure out some other TCP-based protocol, e.g. CoAP. If data transmission is going to happen less frequently than every 10 minutes, HTTP-bridge could be a solution up to 100 M devices. Considering IoT Use Case Adoption Report 2021, MQTT is a good choice for remote asset monitoring, vehicle fleet management, location tracking and on-site track and trace use cases. HTTP is a good choice for IoT based process automation and predictive maintenance use cases.

References

1. Hanes D, Salgueiro G, Grossetete P, Barton R, Henry J (2017). IoT fundamentals: Networking technologies, protocols, and use cases for the internet of things. Cisco Press. Indiana, United States.  

   [Google Scholar]

2. Pierleoni P, Concetti R, Belli A, Palma L (2019). Amazon, Google and Microsoft solutions for IoT: Architectures and a performance comparison. IEEE access. 8: 5455-5470.

   [Crossref] [Google Scholar]

3. Misra S, Mukherjee A, Roy A (2021). “Introduction to IoT”. Cambridge University Press. Cambridge, USA.

   [Crossref]

4. Maurya R, Nambiar KA, Babbe P, Kalokhe JP, Ingle YS, et al (2021). Application of restful apis in iot: A review. Int J Res Appl Sci Eng Technol. 9: 145-151.

   [Google Scholar]

5. Atmoko RA, Riantini R, Hasin MK (2017). IoT real time data acquisition using MQTT protocol. J Phys Conf Ser. 853: 012003.

   [Google Scholar]

6. Amadeo M, Campolo C, Iera A, Molinaro A (2015). Information Centric Networking in IoT scenarios: The case of a smart home," 2015 IEEE International Conference on Communications (ICC), London, UK. 648-653.

   [Google Scholar] [PubMed]

7. Longo E, Redondi AE, Cesana M, Arcia-Moret A, Manzoni P (2020). Mqtt-st: a spanning tree protocol for distributed mqtt brokers. IEEE International Conference on Communications. 1-6.

   [Crossref] [Google Scholar]

8. Ali AA (2018). Constrained application protocol (CoAP) for the IoT. InIoT Seminar, High Integrity System, Frankfurt University of Applied Science. 1-4.