AWS IoT Solutions

The Internet of Things (IoT) offers unparalleled opportunities for every industry to address its business challenges. With the proliferation of devices, there is a need for a solution to connect, collect, store and analyze device data. Amazon Cloud Technologies offers various services that help connected devices easily and securely interact with cloud applications and other devices to meet a variety of user scenarios.

That said, every solution architect in the space knows the capabilities and reliability of the AWS cloud. Migrating or designing an Internet of Things (IoT) solution to the AWS platform allows people to focus on their core business without infrastructure management and monitoring. This will ensure high availability for customers.

Whichever solution is designed, the best platform should be chosen to keep the solution stable, and AWS is one such platform.

There are a few practices to consider when designing an IoT solution with AWS. If the right AWS services are used to meet customer requirements, then the IoT solution can deliver results more securely, reliably and scalable manner.

Designing for reliable large-scale operation

IoT systems must handle the high speed and volume of data devices and gateway capture. Overflow of incoming data is to be expected due to sudden business growth or sometimes due to malicious attacks. The cloud system architecture should be scalable to handle such data in such cases.

The best way is to send the data to a queue, buffer, or real-time in-memory database, which is then stored in storage. This helps enable real-time events and slows down the rate of data insertion to prevent database crashes or slower response times.

The appliance can publish data to AWS Kinesis, or AWS IoT rules can forward data to AWS SQS and Kinesis to store it in time-series stores such as AWS S3, Redshift, DataLake, or Elasticsearch for data storage. These data stores can generate custom control panels or AWS QuickSight control panels.

Routing large data volumes through data pipelines

Using incoming data from device topics directly to a single service can prevent the system from achieving full scalability. Sometimes, this approach can limit system availability on failures and data flooding events.

The AWS IoT Rules Engine is designed to connect end nodes to AWS IoT Core in a scalable manner, but all AWS services have different data flow properties and their respective strengths and weaknesses. All services cannot be used as a single entry point to the system. Sometimes, it can lead to subsequent failures from which there is no recovery.

For example, consider caching (ElastiCache) or queuing (SQS) incoming data for large amounts of data before invoking other services to recover from subsequent failures.

The AWS IoT Rules Engine allows multiple AWS services such as Lambda, S3, Kinesis, SQS, and SNS to be triggered in parallel. Once the IoT system captures data, it enables AWS endpoint nodes (other AWS services) to process and transform the data. This allows data to be stored in multiple data stores simultaneously.

The safest and best way to ensure that all data is processed and stored is to redirect all device subject data to an SNS designed to handle data flood processing and ensure that incoming data is reliably maintained, processed and delivered to the appropriate channel. Multiple SNS topics, SQS queues, and Lambda for different/groups of AWS device topics can be used to make it more scalable.

Before processing, consider storing data in secure storage such as Queue, Amazon Kinesis, Amazon S3, and Amazon Redshift. This approach ensures no data is lost due to message flooding, unwanted exception code, or deployment issues.

Automate device configuration and upgrades

As the business grows and many devices connect to the IoT ecosystem, manual processes such as device configuration, bootstrapping software, security configuration, rule action setup, and device OTA upgrades are not feasible. Minimizing manual interactions during initialization and upgrades is important to save time and cost.

Designing built-in functionality in the device to automate the configuration and leveraging the appropriate tools provided by AWS to handle device configuration and management allows the system to achieve the required operational efficiency with minimal human intervention.

AWS IoT provides a set of features that can be used for bulk import that contain a set of policies that can be integrated with a control panel or manufacturing process where devices can be pre-registered to AWS IoT, and certificates can be installed in the devices. Later, a device pre-assignment flow can declare the device and attach it to the user or any other entity. AWS provides tools to trigger and track device OTA upgrades.

Scalable architecture for custom components

When an IoT system connects to an outside-world device, the scope doesn't end with connecting, controlling and reporting on the device. Consider adopting the latest technologies, such as data science and machine learning, or integrating third-party components into IoT systems, such as IFTTT, Alexa or Google Home. The IoT architecture should ensure that external components can be easily integrated into the solution without performance bottlenecks.

Check offline access and processing

Sometimes it is not necessary to process all machine data in the cloud. In many cases, there is no continuous Internet connection available. For such scenarios, add AWS Greengrass at the edge. Greengrass processes and filters data locally at the edge, reducing the need to send all device data upstream. All data can be captured, held for a limited time, and sent to the cloud on error events or on-demand/requests. If time-series data is required, a periodic process can be scheduled to send device data to the cloud that can be used for future enhancements, such as AWS machine learning models and cloud analytics tools.

Choose the right data storage

IoT systems generate high-speed, high-volume and diverse data. Each IoT device or device topic can have different formats that may not be managed through a single database or similar data store. Architects should be careful when choosing database formats and data stores. Sometimes a single data store works fine, or a mix of data stores for different purposes can help achieve high throughput. Commonly used static data can be stored in ElastiCache, which helps to improve performance. This practice helps to achieve scalability and maintainability of the system.

Filter and transform data before processing

All incoming data to the IoT system may need to be processed or transformed, after which it can be redirected to storage as-is.AWS IoT rules provide operations to redirect messages to different AWS services. The architect should lurk all data in different forms, i.e., it needs to be processed, ignored/static data (e.g., Config), and stored directly.

AWS IoT Service Architecture

AWS IoT helps enable fast device connectivity, secure data ingestion, easy device management, multi-protocol support, and more.