The devops industry has increasingly embraced infrastructure as a code (IaC) over the past few years. Managing your infrastructure in such a way has multiple benefits, such as keeping a history of your infrastructure changes, allowing quick rollbacks and allowing peer review.

There are plenty of tools out there to achieve this goal, such as Terraform, Crossplane, Pulumi and more.In this article, we will focus on how to manage your resources with dependencies and constraints. We use Pulumi to manage our cloud resources, but the principle is the same for all IaC tools.

What is a Pulumi stack?

According to Pulumi documentation, “Pulumi stack is an isolated, independently configurable instance of a Pulumi program.” The cloud resources of each stack are saved in their own state file.

When we change the configuration or the code and apply the change (pulumi up command), Pulumi will compare the new state that was created by the configuration and code with the current saved state.

Pulumi commands run against a specific stack, so if changes are made, all resources in that stack could potentially be affected.

The Current Project

In every Pulumi project, one of the more consequential questions to answer is how it could be separated into different stacks. This topic is mentioned in the pulumi documentation here.The common options are

1. Monolithic - each stack represents a gcp project
2. Micro-stacks - each network resource (network resource is combined by multiple cloud resources) should be in its own stack.

Monolith is simple at first. But, as the code, functionality or the amount of cloud resources increases, it becomes more complex and difficult to maintain. The monolith approach is a good way to start your journey with Pulumi, but when you have a lot of cloud resources, micro stacks are a better solution.

Micro stacks vs Monolithic

To better understand these approaches, assume we are managing the network layer in GCP in a monolith approach. In this approach, dozens of Google load balancers and other network components will be managed in one stack. The risks are:

1. Every change can potentially change another resource in the stack, so each change should be handled with extra care.
2. Having all the resources in one place makes it difficult to understand what is connected to what, and what resources are no longer used.

We decided to look at micro-stacks instead of monoliths to prevent these risks. The micro stack approach can be implemented in many ways. In our case, we decided that each load balancer will have its own stack.

In this way, we can be confident that the scope of the change is a single load balancer and that the change will not affect other load balancers.

The two main things we have to address in this approach is the dependency between shared resources and the ability to run one Pulumi command against multiple load balancers.

Bypassing the Limitations

Using the pulumi automation API, we created a light CLI tool that wraps the Pulumi commands.As shown below, each stack (load balancer) is represented in the configuration file:

hello-world_load_balancer.yaml

backend_services:
- name: hello-world-service
 load_balancing_scheme: EXTERNAL
 health_checks: hello-world-health-check
 backends:
 - max_utilization: 0.8
   name: hello-world-mig
   zone: europe-west1-b
health_checks: []
http_load_balancers:
- name: hello-world-load-balancer
 urlmap:
   default_backend: hello-world-service
 frontends:
   https:
     ip_port:
     - port: 443
       ip_name:  hello-world-ip
     ssl_certificates:
       default_certificate: perimeterx-com
depends_on:
- common-healthcheck

common_healthcheck.yaml

health_checks:
- name: common-health-check
 http_health_check:
   request_path: /alive
   port: 9095

In this example, common_healthcheck.yaml is defining a common health check that is being used by different backend services. There is dependency between the two stacks because the backend service uses a health check that is not defined in its own file.

A dependency list is obtained by the depends_on attribute in each configuration file.

The CLI tool gets the configuration files as arguments and reads their dependencies.

In the case of multiple layers of dependency, the CLI will create a graph of all the dependencies, validate they are not circular and execute them in the correct order.As a result, if changes occur in both stacks that depend on each other, the tool will run them both in the correct order.

The communication between the stacks is performed using pulumi outputs.

This CLI tool is addressing the parallel operation against multiple stacks.

By getting a list of stacks, building the dependency graph as mentioned above, and running the pulumi commands simultaneously, we can run horizontal changes quickly as a built-in functionally.

Code snippets

1. Pseudo code of the main flow

   def func pulumi_up(stack_files):
      graph = build_dependecy_graphs(stacks_file)
      # seperate into steps
      # each step contains multiple stacks
      # the stacks in the step can run in parallel
      # steps should run in the right order
      steps = seperate_into_steps(graph)
      executor = ThreadPoolExecutor(thread_size)
      for step in steps:
          for stack in step:
              stack = pulumi_api.select_stack(stack)
              executor.submit(lambda: stack.up())

2. Handling the dependency using networkx

   def build_dependency_graph(self, current_stack):
     if not self.is_gen_empty(nx.simple_cycles(self.graph)):
         logging.error('there is dependency cycle, exiting')
         exit(1)
     self.graph.add_node(current_stack)
     config = self.read_yaml(current_stack)
     if 'depends_on' in config:
         for dependency_stack in config['depends_on']:
             self.graph.add_edge(dependency_stack, current_stack)
             self.build_dependency_graph(dependency_stack)

3. Create steps from the graph

   def create_steps(self):
     copied_graph = self.graph.copy()
     steps = []
     while copied_graph:
         zero_indegree = [v for v, d in copied_graph.in_degree() if d == 0]
         steps.append(zero_indegree)
         copied_graph.remove_nodes_from(zero_indegree)
     return steps

Summary

From our experience, moving to a micro-stack approach is just a question of time. With the principles above, this transition can be much easier and smoother than previously assumed.