Fun with Terraform Template Rendering
One of the things I do as part of Eiara is write a lot of Terraform, an infrastructure definition language, to provide a sensible baseline cloud instantiation of infrastructure and resources.
I’m quite fond of Terraform as a tool, even though it still has a decent number of weirdnesses and edge cases. If you haven’t seen it you should look at Charity’s Blog about Terraform for a great rundown on those issues. It’s quite powerful, and really works well with how I think to enable me to express exactly what I’m thinking.
Part of what I’m doing requires rendering out JSON templates for use with AWS. This is a pretty normal requirement for doing anything with AWS, from the IAM policies to the ECS task definitions. Lots and lots and lots of JSON.
Lots. (lots)
Specifically, right now I’m trying to do is make a list of dicts, where each dict is the representation of a single module, which then gets jammed together to be inserted as a list in a larger block of JSON.
Straightforward, right?
Well…
Gotchas as a Service
First off, the best way to see what’s going on is to write out what’s going on to disk and look at it. Terraform doesn’t directly let you do that, instead requiring an approximation, with something like:
resource "null_resource" "export_rendered_template" {
provisioner "local-exec" {
command = "cat > test_output.json <<EOL\n${data.template_file.test.rendered}\nEOL"
}
}Note the \n’s in order to make sure the multiline expands properly. Of course, nothing could go wrong with a rogue template here, but I digress.
But, we can write content out to disk, and check that our JSON blobs are working as expected. Great!
That’s Interesting
Next up, template files. We render them with a straightforward block,
data "template_file" "test" {
count = "${length(var.alist)}"
template = "${file("./tpl.json")}"
vars {
variable = "${var.myvar}"
}
}and everything is fine.
Interestingly, files that are rendered by the Terraform template system have access to the full range of functions provided by the Terraform interpolation engine. This means that you can use the file() function from inside a template file.
That’s curious. I’m sure nothing bad could happen there.
Complex Data Types and Templates
Back to trying to render my JSON. The first thing I tried was just to plug a list in, and try to render it inside the template, much like this. variable “alist” { type = “list” default = [1,2] }
data "template_file" "test" {
template = "${file("./tpl.json")}"
vars {
alist = "${var.list}"
}
}Unfortunately, Terraform doesn’t, as of 0.8.7, let you pass complex types into the template renderer, so that doesn’t work.
However, if we use join(",", var.alist), it’ll render much as we expect it to for numbers.
{
"list":[1,2]
}What about if we use strings?
variable "alist" {
type = "list"
default = ["a","b"]
}Output:
{
"list":[a,b]
}Well, that breaks. But! We have the jsonencode() function, which returns blocks of JSON. Great! We can render our list arbitrarily!
List of Strings of Rendered JSON
But the goal here is to drop a list of rendered blobs of JSON into our template. How does that hold up with jsonencode ?
variable "alist" {
type = "list"
default = [<<EOF
{"foo": "bar"}
EOF
,"b"]
}Output:
{
"list":["{\"foo\": \"bar\"}\n","b"]
}Hm. That’s not good, but, surprisingly, we can use HEREDOCs inside a list declaration.
Neat. But, I digress.
What about a nested map? Will that work?
variable "amap" {
default = {
foo = {
baz = "bar"
beez = ["a"]
}
}
}Output:
Errors:
* jsonencode: map values must be strings in:
${jsonencode(var.amap)}Hrm, not directly.
And we can’t pass complex types into templates.
We could use string literals, but then we’re not able to pass in an arbitrary number of elements through our list. Also, since we’re expecting to return rendered bits of JSON from our modules, this is just going to wrap things in strings, which isn’t what we want anyway.
Okay, What About
So if all I want to do is make a list of dicts, I should be able to render to JSON the dict initially, and then just join the properly rendered JSON blobs with a comma.
Let’s test that.
data "template_file" "test" {
count = "${length(var.alist)}"
template = "${file("./tpl.json")}"
vars {
alist = "${jsonencode(element(var.alist, count.index))}"
}
}
resource "null_resource" "export_rendered_template" {
provisioner "local-exec" {
command = "cat > test_output.json <<EOL\n${join(",\n", data.template_file.test.*.rendered)}\nEOL"
}
}Output:
{
"list":"a"
},
{
"list":"b"
}Okay that’s really close! We’re rendering into templates and then just joining it together with a ,\n and it appears to be what we want.
So in order to get it to look right, we’ll need to wrap it in an additional template_file to add the [] pair that we need to have a proper list of dicts, such as
data "template_file" "test" {
count = "${length(var.alist)}"
template = "${file("./tpl.json")}"
vars {
alist = "${jsonencode(element(var.alist, count.index))}"
}
}
data "template_file" "test_wrapper" {
template = <<JSON
[
$${list_of_dicts}
]
JSON
vars {
list_of_dicts = "${join(",\n", data.template_file.test.*.rendered)}"
}
}
resource "null_resource" "export_rendered_template" {
provisioner "local-exec" {
command = "cat > test_output.json <<EOL\n${data.template_file.test_wrapper.rendered}\nEOL"
}
}Output:
[
{
"list":"a"
},
{
"list":"b"
}
]That’s really close! The indentation is a bit off, but Python can read it!
Complexity
This is obviously a bit of a weird, complex case. By trying to hide the abstractions of JSON blocks that represent, in my case, ECS Container Definitions, I’m requiring other places in my code to craft the correct JSON blobs.
But it also feels like good programming practise to build abstractions on top of things like container definitions and provide a cleaner interface to the components I’m working with. And because I’m building two abstractions, one for the container definition and one for the task definition, I can hide the nesting of templates and, in the end, pass a list of module outputs to a module and have it do The Right Thing.
And that feels right.