Testing terraform with moto, part 2 (An Interlude)
March 28, 2022
In my last post I started to set up the terraform and moto environment. In this post, we’ll attempt to connect an existing module up to moto server.
I had previously exported and converted the terraform output of builder to standard terraform HCL, and decided to build upon that work to:
- Give the scenario a real-world setup
- Save myself time trying to write it from scratch.
There is a branch called refactor-into-a-module That I used as the base version. Then I used the basic config in our last post as a test terraform module utilising the module. The code in that repo is, unfortunately, stuck on terraform 0.11, so I needed to update quite a bit of it to bring it up to date to latest terraform coding pracices. In the end, we had something like this
1// setup provider for localstack
2provider "aws" {
3 region = "us-east-1"
4 access_key = "test"
5 secret_key = "test"
6 s3_use_path_style = true
7 skip_credentials_validation = true
8 skip_metadata_api_check = true
9 skip_requesting_account_id = true
10
11 endpoints {
12 ec2 = "http://localhost:5000"
13 eks = "http://localhost:5000"
14 iam = "http://localhost:5000"
15 s3 = "http://localhost:5000"
16 }
17}
18
19module "kubernetes-aws" {
20 source = "../../../modules/kubernetes-aws"
21 cluster_name = "kubernetes-test"
22 cluster_version = "1.19"
23 project = "terraform-testing"
24 environment = "test"
25 min_workers = "1"
26 max_workers = "6"
27 desired_workers = "3"
28 instance_type = "t3.large"
29}
I then worked in a terraform init and terraform plan, trying to correct any issues I could see with the code. In the end, I needed to replace some hardcoded VPCs, a call to AWS for EKS ec2 images (instead, make sure the call returns a moto mocked image), and some small tweaks to changes in the resources available in the aws plugin.
In the end, I had something like this:
1// setup provider for localstack
2provider "aws" {
3 region = "us-east-1"
4 access_key = "test"
5 secret_key = "test"
6 s3_use_path_style = true
7 skip_credentials_validation = true
8 skip_metadata_api_check = true
9 skip_requesting_account_id = true
10
11 endpoints {
12 ec2 = "http://localhost:5000"
13 eks = "http://localhost:5000"
14 iam = "http://localhost:5000"
15 }
16}
17
18# query moto to get the default VPC and subnet
19data "aws_vpc" "default" {
20}
21data "aws_subnets" "default" {
22 filter {
23 name = "vpc-id"
24 values = [data.aws_vpc.default.id]
25 }
26}
27
28module "kubernetes-aws" {
29 source = "../../../modules/kubernetes-aws"
30 cluster_name = "kubernetes-test"
31 cluster_version = "1.19"
32 project = "terraform-testing"
33 environment = "test"
34 min_workers = 1
35 max_workers = 6
36 desired_workers = 3
37 instance_type = "t3.large"
38
39 is_testing = true # added to signal which image filter to look for
40 vpc_id = data.aws_vpc.default.id # to replace the hardcoded VPC ids
41 subnets = data.aws_subnets.default.ids # ... and subnets with moto ones
42}
At this point, in theory the module should apply cleanly to AWS infrastructure. Unfortunately, I ran into an issue with moto:
1│ Error: error creating EKS Cluster (kubernetes-test): ResourceInUseException: Cluster already exists with name: kubernetes-test
2│ {
3│ RespMetadata: {
4│ StatusCode: 409,
5│ RequestID: ""
6│ },
7│ ClusterName: "kubernetes-test",
8│ Message_: "Cluster already exists with name: kubernetes-test"
9│ }
10│
11│ with module.kubernetes-aws.aws_eks_cluster.main,
12│ on ../../../modules/kubernetes-aws/main.tf line 206, in resource "aws_eks_cluster" "main":
13│ 206: resource "aws_eks_cluster" "main" {
14│
Bringing the module down the bare minimum:
1// setup provider for localstack
2provider "aws" {
3 region = "us-east-1"
4 access_key = "test"
5 secret_key = "test"
6 s3_use_path_style = true
7 skip_credentials_validation = true
8 skip_metadata_api_check = true
9 skip_requesting_account_id = true
10
11 endpoints {
12 ec2 = "http://localhost:5000"
13 eks = "http://localhost:5000"
14 iam = "http://localhost:5000"
15 s3 = "http://localhost:5000"
16 }
17}
18
19
20# get the default VPC out of moto
21data "aws_vpc" "default" {
22}
23data "aws_subnets" "default" {
24 filter {
25 name = "vpc-id"
26 values = [data.aws_vpc.default.id]
27 }
28}
29
30#create a basic IAM role
31resource "aws_iam_role" "master" {
32 name = "test"
33 assume_role_policy = "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Action\": \"sts:AssumeRole\", \"Effect\": \"Allow\", \"Principal\": {\"Service\": \"eks.amazonaws.com\"}}]}"
34}
35
36#create a basic security group
37resource "aws_security_group" "master" {
38 name = "test"
39 vpc_id = data.aws_vpc.default.id
40
41 egress {
42 cidr_blocks = [
43 "0.0.0.0/0",
44 ]
45
46 from_port = 0
47 to_port = 0
48 protocol = -1
49 }
50
51 description = "test"
52}
53
54# create a basic eks cluster
55resource "aws_eks_cluster" "main" {
56 vpc_config {
57 subnet_ids = data.aws_subnets.default.ids
58
59 security_group_ids = [
60 "${aws_security_group.master.id}",
61 ]
62 }
63
64
65 version = "1.21"
66 role_arn = "${aws_iam_role.master.arn}"
67 name = "test"
68}
and running terraform apply with the environment variable TF_LOG=debug we can see where things go wrong:
12022-03-28T07:37:38.803+0100 [DEBUG] provider.terraform-provider-aws_v4.5.0_x5: [aws-sdk-go] <lengthy json>: timestamp=2022-03-28T07:37:38.803+0100
22022-03-28T07:37:38.803+0100 [DEBUG] provider.terraform-provider-aws_v4.5.0_x5: [aws-sdk-go] DEBUG: Unmarshal Response eks/CreateCluster failed, attempt 0/25, error SerializationError: failed decoding JSON RPC response
3 status code: 200, request id:
4caused by: JSON value is not a list (map[string]interface {}{}): timestamp=2022-03-28T07:37:38.803+0100
52022-03-28T07:37:38.862+0100 [DEBUG] provider.terraform-provider-aws_v4.5.0_x5: [aws-sdk-go] DEBUG: Retrying Request eks/CreateCluster, attempt 1: timestamp=2022-03-28T07:37:38.862+0100
The try then fails, as the (fake) cluster was already created.
From the error output, it would seem terraform code is expecting something in that json response to be a list, but is finding it as a map.
Sure enough, the moto codebase shows that it is returning a dict():
1class Cluster:
2 def __init__(
3 self,
4 name,
5 role_arn,
6 resources_vpc_config,
7 region_name,
8 aws_partition,
9 version=None,
10 kubernetes_network_config=None,
11 logging=None,
12 client_request_token=None,
13 tags=None,
14 encryption_config=None,
15 ):
16 if encryption_config is None:
17 encryption_config = dict() # THIS LINE
18 if tags is None:
19 tags = dict()
Where as aws’s API documentation says it’s an array andaws-sdk-go expects a list in the response.
I’ve posted a bug request here, hopefully this will be ironed out upstream.
In the next post, I’ll explore a temporary fix for this, and then using terratest to set out examples and tests. Until next time.
- Scott