Polymorphic Lambdas with StepFunctions

When you use step functions with lambda microservices, it may be necessary in some cases to execute specific versions of lambdas. In short, lambda microservices with polymorphic behaviour.

This kind of requirement could be common in Machine Learning. This bit I am not entirely sure as I am still new to ML. However, this problem is interesting as the alternative would require you to run different instances of same lambdas in your environment to cater to different sources. We have come across such a requirement where the source dictates which algorithms to trigger in a flow.

The algorithms are mostly similar with a few tweaks so depending on each source the function executed is selected based on the context or source.

Here are some steps you can easily follow to solve this problem :-

The following two actions could be part of your deploy step. The steps below create an alias or qualifier with which you can invoke the specific versions.

Create Version for each Lambda

VERSION=$(aws lambda list-versions-by-function \     
    --function-name microservice_1 \     
    --no-paginate \     
    --query "max_by(Versions, &to_number(to_number(Version) \
      || '0'))" | jq -r '.Version')

Create an Alias

aws lambda create-alias \     
    --function-name microservice_1 \     
    --name v1 \     
    --function-version $VERSION \     
    --description " "

Source to Version Mapping

In your database you could have a mapping of input source to version map for subsequent microservice versions to execute. So below the versions match the qualifier that you configured as alias earlier.

source 1   ->   {
                    microservice 1 -> v1
                    microservice 2 -> v6
                    microservice 3 -> v4
                }
source 2   ->   {
                    microservice 1 -> v2
                    microservice 2 -> v5
                    microservice 3 -> v7
                }

Create version map as part of the payload

For example if you had a set of versions to execute as below for a given source 1.

{
    "microservice_1" : "v1",
    "microservice_2" : "v6",
    "microservice_3" : "v4"
}

Just pack this as part of your payload

{
......
  "release_versions" : {
    "microservice_1" : "v1",
    "microservice_2" : "v6",
    "microservice_3" : "v4"
  }
}

Configure your State in Step Functions

In the step functions you can use the dynamically injected versions block to trigger specific versions of each microservice with the qualifier variable.

{
  ....
  "States": {
    ....
    "microservice 1": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "microservice_1",
        "Qualifier.$": "$.release_versions.microservice_1"
      },
      "Next": "microservice 2"
    },
    "microservice 2": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "microservice_2",
        "Qualifier.$": "$.Payload.release_versions.microservice_2"
      },
      "Next": "microservice 3"
    },
    "microservice 3": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "microservice_3",
        "Qualifier.$": "$.Payload.release_versions.microservice_3"
      },
      "End": true
    }
  }
}

That’s it! Correct versions of the lambda’s will execute.

Note the .Payload in the second and third states. This is because when you execute a lambda with an alias, it wraps up the output with meta-data information. The metadata contains execution information such as which version was used to execute the lambda and more. So the subsequent lambdas need the additional .Payload to access the actual payload returned by the previous step.