The Particular Finest

More Fun with Terraform Templates

So you may have noticed last time that I said I’m trying to create complex JSON objects from within Terraform.

In this case, I really want to be able to create an AWS ECS container definition, which looks a bit like this (copied from the AWS docs, here)

{
 "name": "wordpress",
 "links": [
   "mysql"
 ],
 "image": "wordpress",
 "essential": true,
 "portMappings": [
   {
     "containerPort": 80,
     "hostPort": 80
   }
 ],
 "memory": 500,
 "cpu": 10
}

The important part for me here is making a module to create these JSON blocks. This will let me keep all the variables in Terraform variable files, and ensures that I can interrogate the state file as to what variables are set, and for what container definition.

Ideally, I want the declaration to look something like this

module "container_definition" {
source = "./ecs_container_defintion"

name = "container"
image = "hello-world"
essential = "true"
memory = 500
cpu = 10
port_mappings = [
  {
    "containerPort": 80,
    "hostPort": 80
  }
]
}

So, we only have three basic pieces of data to worry about here:

1. Simple key-value associations
2. Array of strings
3. Arrays of maps

As we looked at last time, the jsonencode function can’t deal with an array of maps (or any complex datatype), so we have to unpack this manually.

But, we also can’t use a jsonencode for the basic data pieces either, because making a map that we then encode means we’d end up with a JSON string that we couldn’t expand with the complex data types we need to create.

So that won’t work.

What will work, however, is the bit we used last time, specifically the join(",\n", var.list) that we used. However, instead of using a variable directly, we can instead create the list on the fly using the list() function from Terraform.

Layer 1

That’s set the scene on what we want and how we’ll get it to work. Let’s dig into what it’d look like.

I’m going to skip the variable declarations this time around, and focus on just the data declarations and the resulting JSON blocks.

To start, let’s just have a basic module call, like this

module "container_definition" {
  source = "./ecs_container_definition"
  name = "container"
  image = "hello-world"
  essential = true
}

Three things, completely straightforward. Should be easy.

So, going with our dynamic list and join, what will the template look like? Probably something like this

data "template_file" "_final" {
template = <<JSON
{
  $${val}
}
JSON
vars {
  val = "${join(",\n    ",
      list(
        "${jsonencode("name")}: ${jsonencode(var.name)}",
        "${jsonencode("image")}: ${jsonencode(var.image)}",
        "${jsonencode("essential")}: ${var.essential ? true : false }",
        )
    )}"
}
}

So here’s where it’s starting to get a bit, well, not great. As you’ve noticed, each key in the list has to be run through jsonencode, to ensure that it’s properly quoted. The values have to be wrapped in quotes as well, so they’re encoded.

Because we don’t want a list of single-key JSON object strings, we can’t just encode as a list of maps.

var.essential is interesting, as well. Passing a boolean value like true above gets converted to 1 by the module process, so here we just cast it back.

This will probably fail miserably if you pass in the "false" string, instead of the false boolean.

Finally, when we render it, we get

{
  "name": "container",
  "image": "hello-world",
  "essential": true
}

Which looks perfect! Exactly what we want.

Links in

Next, let’s add the links array. This one should be easy, because it’s just a list of strings, and we can rely entirely on jsonencode.

"${jsonencode("links")}: ${jsonencode(var.links)}"

Easy. And the output is right, too

"links": ["mysql"]

Port of Call

Next, we’ll add in the port mapping section, the complex part, the array of maps, which is the first point where we need to break out a second template_file to handle the rendering. This is specifically so we can use the count construct to iteratively jsonencode the elements of the list, and then wrap the entire contents in [].

This is going to look something like

data "template_file" "_port_mapping" {
  count = "${length(var.port_mappings)}"
  template = "$${val}"
  vars {
    val = "${jsonencode(var.port_mappings[count.index])}"
  }
}

We can be pretty dense here, as all we’re trying to ensure is that each element of our array has been rendered by jsonencode, and doesn’t require additionally complex actions.

Adding it to our list would be

"${jsonencode("portMappings")}: [
   ${join(",\n", data.template_file._port_mapping.*.rendered)}
]"

which gives us the output we’re looking for:

"portMappings": [
   {"containerPort":"80","hostPort":"80"}
]

Great!

Okay, so, what if we leave things off? image and name aren’t optional, so we just don’t provide a default and let the Terraform compiler handle that case. essential isn’t an essential field, I think we should be able to drop that successfully. Let’s do that.

Hm

Errors:

* __builtin_StringToBool: strconv.ParseBool: parsing "": invalid syntax in:

Well,

That’s not good. That’ll be the var.essential ? : section, where we try to cast an int into a boolean.

So we’ll need to detect if we’re passing in the default empty string, and do something useful based on that.

But that’s easy! We’ll just another ternary to test it! Something like this

"${ var.essential != "" ? "${jsonencode("essential")}: ${var.essential ? true : false }" : "something" }",

and then we go again, and

Errors:

* __builtin_StringToBool: strconv.ParseBool: parsing "": invalid syntax in:

…

oh

it’s evaluating it… twice…

Hrm.

Okay. We can solve this. How about

...
"${var.essential != "" ? data.template_file.essential.rendered : ""}",
...

data "template_file" "essential" {
  template = "$${jsonencode("essential")}: $${val ? true : false}"
  vars {
    val = "${var.essential != "" ? var.essential : "false"}"
  }
}

Eesh. That’s not great. It works, but, yeah, not very well. The first evaluation always takes place, even if the other branch in the comparison is taken. This means that, no matter what, I have to create the essential template node, even if essential is undefined, to pull off this effect.

You may be asking “why is she even trying to cast things to true or false? JSON says it’ll just work.”

And the answer is because Terraform tries to be clever, and turns the JSON blob into a struct. Which is strictly typed to expect a bool.

Which means it complains loudly at anything that’s not a literal true.

Fortunately, CPU and Memory should be easy, we can just test if they’re defined inline easily, such as

"${var.cpu != "" ? "${jsonencode("cpu")}: ${var.cpu}" : "" }",

Collapse the List

Rendering out an empty string, "", does have one negative side effect, in that we end up with a JSON block that’s invalid. However, there was a reason I picked the empty string as my return value, and that’s the compact() function in Terraform.

compact() takes an array, and strips out all the items that are empty, so changing cpu above, for example, means the entire

"cpu": 10

line just won’t render if cpu isn’t defined, which is perfect for our “dynamic” goal.

Back to Port

Okay, so, back to port mappings.

Unfortunately, due to the complexity of the operation, we need to do the same thing we did with the essential entry to ensure that it is a bool, and break it into its own template_file, like this

data "template_file" "_port_mappings" {
  template = <<JSON
"portMappings": $${val}
JSON
  vars {
    val = "${join(",\n", data.template_file._port_mapping.*.rendered)}"
  }
}

and address it in the final render like so

"${length(var.port_mappings) > 0 ?  data.template_file._port_mappings.rendered : ""}"

And, lo and behold, it renders correctly.

This has “bad idea” written all over it

Of course, the proof is in the pudding: Will AWS accept this as a valid task definition?

After removing links, our final rendered block is

{
  "name": "container",
  "image": "hello-world",
  "cpu": 10,
  "memory": 500,
  "essential": true,
  "portMappings": [
    {"containerPort":"80","hostPort":"80"}
   ]
}

which looks agreeably correct, to me, but it’s not me that must be agreeable, but Terraform and AWS.

And the answer is

…

no.

At some point in this, our map had the port values changed from integers into strings, and Terraform doesn’t cast from strings when deserialising the JSON blob.

sigh fine.

So, each portMapping has three elements: a hostPort, a containerPort, and an optional protocol, where protocol can be either ”tcp” or ”udp”.

Because it’s two different kinds of things, we’re going to need another compacting list render step.

OKAYFINE.

OKAYFINE

So, we need more control over the port mapping render. We already broke it out into its own template_file block, so let’s start there.

Because of how we’re using the count.index to create a terraform node array, we’ll have to check for enumeration here. We can’t use the same trick we’re using in the main renderer, where we collapse a list using join, at least not in the same way.

Actually

Maybe we can.

Because port_mappings is an array of maps, we can use element() to pull a variable out of the map during our iteration, and use the default to return a “” in the case that it’s not present.

Which we can then use as our list elements

Which we can then collapse into just the elements that exist

which we can turn into our rendered dict! Like this!

data "template_file" "_port_mapping" {
  count = "${length(var.port_mappings)}"
  template = <<JSON
$${join(",\n", 
  compact(
    list(
    hostPort == "" ? "" : "$${ jsonencode("hostPort") }: $${host_port}",
    "$${jsonencode("containerPort")}: $${container_port}",
    protocol == "" ? "" : "$${ jsonencode("protocol") }: $${jsonencode(protocol)}"
    )
  )
)}
JSON
  vars {
    host_port = "${ lookup(var.port_mappings[count.index], "hostPort", "") }"
    # So that TF will throw an error - this is a required field
    container_port = "${ lookup(var.port_mappings[count.index], "containerPort") }"
    protocol = "${ lookup(var.port_mappings[count.index], "protocol", "") }"
  }
}

Isn’t it beautiful.

Unto the Breach

Okay, our final rendered output looks like

{
  "name": "container",
  "image": "hello-world",
  "cpu": 10,
  "memory": 500,
  "essential": true,
  "portMappings": [
{
"hostPort": 80,
"containerPort": 80
}

]

}

We’re getting a decent amount of spurious whitespace at this point, but we’re just going for a proof of concept. We can clean that up later.

Again, the proof is in the pudding. Will Terraform take this snippet?

YES!

Next Steps

yes

At this point, this module is a very barebones implementation, and doesn’t support the majority of options that the ECS task definition supports, but now that we have a reasonably complete implementation for the basics it should be relatively straightforward to fill out the rest of available options.

Just Because You Can

This is probably an excellent example for that old axiom of, just because you can’t doesn’t mean you should. At the same time, it provides a considerably cleaner interface to the user to work with a container definition, which is an important win.

More than anything, I think it’s amazing to see how you can bend a tool that clearly isn’t designed to generate dynamic JSON into generating dynamic JSON.

Update: Jan 27, 2018

The module described in this post (and the previous post) is publicly on Github! MIT license, and you can clone it here, if you’d like to do that sort of thing!

• #devops
• #terraform
• #tech
• #tools