AWS IoT — Part 2

THE THING SHADOW

Courtesy John Carpenter’s horror movie “The Thing”

When you read “The Thing” it always sounds scary because it reminds of the horror movie. The Thing Shadow sounds similar. I was amused when AWS called “devices” as “things”. Hopefully it will not put you off from reading further. Apologies for the dramatic poster! The Thing Shadow is not as scary.

In part 1 we discussed how to get the message from the device across to AWS IoT core and vice versa. In this blog we will delve deeper into the concept of device shadows.

In any IoT system you would have devices sending stream of status information. At any given point the current state of the device is quite important in terms of displaying on the dashboard or evaluating a change of device state to generate an alert.

You would normally do this by storing device status logs and then querying the latest state of each input or output for any given device. This is usually expensive in terms of compute query as you may have billions of logs. Querying is certainly not the most efficient way if you are going to regularly hit the database where the information is not even likely to change.

You could store the current state of the device in a cache when new updates come in. This would be a workable solution. But if you wanted an out of the box solution that does most of the state handling of things or devices then you are in luck.

The Thing Shadow service makes it simple to keep current state of inputs and also provides a mechanism to handle outputs easily. The architecture was not obvious to me to begin with.

Let us look at the Thing Shadow as a as a black box. We need not understand how it works but rather what it can do for you and how to use it.

The diagram below explains the flow of messages. I have simplified topic names so it is easy to read and easy to get a conceptual understanding of the this service.

At high level the shadow service comprises several topics. Each topic serves a mechanism to interact with the service either to send a message to it or receive a message from it.

So here is what the flow looks like in simple terms to get our conceptual understanding first.

1. Let’s imagine a device that has one input and one output wants to interact with IoT Core. Let us call it In0 and Out0.
2. This thing wants to send an update to Shadow service. It sends In0=OFF to the “update” topic.
3. If the message delivery is successful and accepted the shadow service responds on “update/accepted” topic.
4. If the message is not accepted the service responds on “update/rejected”.
5. The In0=OFF also gets updated on another topic called the “update/documents”. The JSON here also holds the previous state of In0.
6. Now the application on the other side of IoT core may want to know the status of In0. So it can interact with the Shadow Service by sending a blank message on “get” topic. Shadow service will respond with latest state info on “get/accepted” topic. If the get request gets rejected the message response is on “get/rejected” topic.
7. Now let’s say the application wants to change the status of Out0 to ON. It sends the Out0=ON to the update topic. This is normally called the desired state for outputs. This is because you would desire this state change but it would only take effect once the thing accepts it and applies this change.
8. The shadow service sends the delta i.e. the difference between the current state and the desired state to an “update/delta” topic.
9. A device could simply subscribe to this topic to know what to do to change its Out0 state for example.

Ok so now lets look at some scenarios and example message set per scenario to get a good understanding of the above flow.

Scenario 1 : Device sends updated status to update topic

Message sent on $aws/things/1001/shadow/update

{
  "state": {
    "reported": {
      "In0": "OFF"
    }
  }
}

Received on $aws/things/1001/shadow/update/accepted

{
  "state": {
    "reported": {
      "In0": "OFF"
    }
  },
  "metadata": {
    "reported": {
      "In0": {
        "timestamp": 1591281321
      }
    }
  },
  "version": 13,
  "timestamp": 1591281321
}

Received on $aws/things/1001/shadow/update/documents

{
  "previous": {
    "state": {},
    "metadata": {},
    "version": 12
  },
  "current": {
    "state": {
      "reported": {
        "In0": "OFF"
      }
    },
    "metadata": {
      "reported": {
        "In0": {
          "timestamp": 1591281321
        }
      }
    },
    "version": 13
  },
  "timestamp": 1591281321
}

Scenario 2 : Device sends malformed status message to update topic

Incorrect JSON message sent to $aws/things/1001/shadow/update

{
    "state": {
        "reported": {
            "In0": "OFF"
        }
    
}

Message received on $aws/things/1001/shadow/update/rejected

{
  "code": 400,
  "message": "Payload contains invalid json"
}

Scenario 3 : Device changes In0 status to ON

Message sent to $aws/things/1001/shadow/update

{
  "state": {
    "reported": {
      "In0": "ON"
    }
  }
}

Message received on $aws/things/1001/shadow/update/accepted

{
  "state": {
    "reported": {
      "In0": "ON"
    }
  },
  "metadata": {
    "reported": {
      "In0": {
        "timestamp": 1591281864
      }
    }
  },
  "version": 14,
  "timestamp": 1591281864
}

Notice the message received on documents topic has both previous and current state info.

Message received on $aws/things/1001/shadow/update/documents

{
  "previous": {
    "state": {
      "reported": {
        "In0": "OFF"
      }
    },
    "metadata": {
      "reported": {
        "In0": {
          "timestamp": 1591281321
        }
      }
    },
    "version": 13
  },
  "current": {
    "state": {
      "reported": {
        "In0": "ON"
      }
    },
    "metadata": {
      "reported": {
        "In0": {
          "timestamp": 1591281864
        }
      }
    },
    "version": 14
  },
  "timestamp": 1591281864
}

Scenario 4 : Application wants to change Out0

Message sent to $aws/things/1001/shadow/update

{
  "state": {
    "desired": {
      "Out0": "ON"
    }
  }
}

Message received on $aws/things/1001/shadow/update/accepted

{
  "state": {
    "desired": {
      "Out0": "ON"
    }
  },
  "metadata": {
    "desired": {
      "Out0": {
        "timestamp": 1591282214
      }
    }
  },
  "version": 15,
  "timestamp": 1591282214
}

Message received on $aws/things/1001/shadow/update/documents

{
  "previous": {
    "state": {
      "reported": {
        "In0": "ON"
      }
    },
    "metadata": {
      "reported": {
        "In0": {
          "timestamp": 1591281864
        }
      }
    },
    "version": 14
  },
  "current": {
    "state": {
      "desired": {
        "Out0": "ON"
      },
      "reported": {
        "In0": "ON"
      }
    },
    "metadata": {
      "desired": {
        "Out0": {
          "timestamp": 1591282214
        }
      },
      "reported": {
        "In0": {
          "timestamp": 1591281864
        }
      }
    },
    "version": 15
  },
  "timestamp": 1591282214
}

Message received on $aws/things/1001/shadow/update/delta

{
  "version": 15,
  "timestamp": 1591282214,
  "state": {
    "Out0": "ON"
  },
  "metadata": {
    "Out0": {
      "timestamp": 1591282214
    }
  }
}

Scenario 5 : Find current status

Message sent to $aws/things/1001/shadow/get

{}

Message received on $aws/things/1001/shadow/get/accepted

{
  "state": {
    "desired": {
      "Out0": "ON"
    },
    "reported": {
      "In0": "ON",
      "Out0": "ON"
    }
  },
  "metadata": {
    "desired": {
      "Out0": {
        "timestamp": 1591282214
      }
    },
    "reported": {
      "In0": {
        "timestamp": 1591281864
      },
      "Out0": {
        "timestamp": 1591282702
      }
    }
  },
  "version": 16,
  "timestamp": 1591282751
}

In the next part we will set up correct policies for the device to interact with shadow service and create actions for input messages by triggering a lambda and alerts via email with SNS for exception reporting.

AWS IoT — Part 1