Wing Your Way to Better Infrastructure with Compiler Plugins

Customizations below the abstraction line with Wing compiler plugins.

Okay, so you have decided to write your amazing application in Wing. You have enjoyed the benefits of Wing's cloud-oriented programming model and high level abstractions. Everything works great, the queues are queuing, the functions are functional, and the buckets are filling. You are ready to hand this application off to the ops team for deployment when suddenly, you are told there is a problem: the infrastructure doesn't comply with your organizations cloud excellence requirements.

Susan, who is an underappreciated, and sleep deprived platform engineer, tells you that your taggable infra resources must adhere to a rigorous tagging convention. She continues to tell you that all buckets must have versioning and replication enabled. You also gather that she was probably going out for drinks later, since she kept going on about her security group and ciders.

Before you take to Twitter and post a long thread about how Wing is not enterprise ready, you recall the tech lead (we will call him Greg) who gave a presentation about Wing at your organization's last Cloud Center of Excellence (CCoE) meeting. Greg assured everyone that they would be able to use Wing and only focus on the functional aspects of their cloud applications. He said this would be made possible by leveraging the organization's custom Wing plugins. So now all that remains is, to figure out what a Wing plugin is.

Welcome to the Wing Plugin System

The Wing SDK is hard at work abstracting away the non-functional concerns of your cloud application. Which is great, now you can focus on the business logic of your application and not even care about what cloud this code will run on. However, these abstractions only solve a piece of the puzzle that is the cloud compiler. Inevitably with any production grade deployment, we will need a way to customize the compilation output to meet business requirements. Whether they be security, compliance, or cost optimizations these scenarios will require drilling down bellow the abstractions and into the compiler.

This is where the Wing plugin system comes in as the first steps to opening up hooks into the Wing compilation process. By using these plugin hooks Wing is still able to decouple the functional and non-functional concerns of our applications. Think of it as the SDK handles all functional concerns such as queues, functions, and buckets, while the plugin system handles the non-functional concerns such as encryption, versioning, and security groups.

The plugin system is boosting the Wing toolchain into the next level of cloud development. Unlocking the ability for teams to solve complex real world problems in Wing without compromising their organizations cloud principles. Actually, the plugin system enables organizations to double down and enforce their cloud principles without slowing down innovation.

But Why?...

I think the "why" is important to talk about for a moment. Why should developers not care about the non-functional requirements of their application when writing code? The answer in my opinion is not that developers should not care about it, it's that they don't want to care about it in most cases. Developers want to focus on innovations and pushing boundaries, not be shackled by the low level details of the cloud. This has been true through the history of software development, that we build abstraction layers on-top of implementation details. Most developers don't want to understand the inner workings of file systems and how they differ between operating systems. We just want to be able to read and write files, thus we have file system abstractions. Then if we need to handle special cases based on operating systems, or CPU architectures we expect the abstraction to give us a way to do that, without rewriting our entire code base.

Though I think it's worth noting that the purpose of the abstraction is not to hide implementation details and make cloud application development more vague, but rather unlock new mental models that drive innovation.

"Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise" - Edsger Dijkstra

This is the "why" for our plugin system in Wing, it's intended to be a mechanism that supports teams to be more precise in their cloud applications. If the SDK unlocks this semantic level of thought, then the plugin system protects it.

The Basics of Compiler Plugins

The plugin system is a simple and powerful way to customize the compilation output of your Wing application. It's comprised of a series of hooks that are called at various stages of the compilation process. In our initial release of the plugin system we have made available 3 hooks: preSynth, postSynth and validate. There are additional hooks that are currently in the think tank, but we will save those for another blog post and focus on what we have available today.

To write a plugin all you need is to implement a JavaScript file, in which you can export one or all of the compiler hooks. Once your plugin is written, you can use the --plugin flag in the wing cli to include it in the compilation process.

wing compile -t tf-aws my-app.w --plugin my-plugin.js

The preSynth hook is executed after the construct tree has been initialized but before the code has been synthesized to produce deployment artifacts. In which our plugins have the opportunity to add and mutate resources in the construct tree.

The postSynth hook's execution is right after synthesis has been completed, and provides a means in which we can manipulate the deployment artifacts (Terraform Config, CloudFormation template, etc).

The validate hook is only executed after all compilation and synthesis has been completed. This is important as this hook is meant to serve as a way to examine and validate deployment artifacts without concerns that some later process will mutate them.

Plugins In Action

No blog post on a new feature would be complete without a walk-through :) So lets walk through the process of writing our own plugin. Not just any plugin though, but one that will help our favorite underappreciated platform engineer, Susan. As more teams have started adopting Wing in her organization she has realized that she can make use of plugins to help teams meet requirements for deploying applications into the cloud, without asking them to rewrite their code.

She has identified a common use case where her organization has deployed an IAM role into every AWS Account using nested stacks. The existence of this role as a permission boundary for all IAM roles is enforced through AWS organization SCPs (Service Control Policies). Thus, without it no IAM role can be created in the account, this is a cause of friction for teams that want to get their Wing applications deployed quickly. (whew thats a whole lot of things a developer should not have to care about)

She has decided to write a plugin that implements 2 hooks preSynth and validate. During the preSynth hook, she wants to add the required permission boundary to all IAM roles in the construct tree. Then she intends to validate the existence of the permission boundary on all roles during the validate hook. This way teams can know their app will fail to deploy at compile time rather than deploy, making for faster feedback loops.

Susan starts with writing the bare necessities of a plugin. She creates a file named permission-boundary-compliance.js and adds the following code:

// Add permission boundary to all IAM roles
exports.preSynth = function (app) { }

// Validate that all IAM roles have a permission boundary
exports.validate = function (config) { }

She plans to make use of a concept from CDKTF known as Aspects to traverse the construct tree and add the permission boundary. She can safely use this since she knows her intended target will be Terraform for AWS.

const iam_role = require("@cdktf/provider-aws/lib/iam-role");
const cdktf = require("cdktf");

class PermissionBoundaryAspect {
  constructor(permissionBoundaryArn) {
    this.permissionBoundaryArn = permissionBoundaryArn;
  }

  visit(node) {
    if (node instanceof iam_role.IamRole) {
      node.permissionsBoundary = this.permissionBoundaryArn;
    }
  }
}

// Add permission boundary to all IAM roles
exports.preSynth = function (app) {
  if (!process.env.PERMISSION_BOUNDARY_ARN) {throw new Error("env var PERMISSION_BOUNDARY_ARN not set")}
  cdktf.Aspects.of(app).add(new PermissionBoundaryAspect(process.env.PERMISSION_BOUNDARY_ARN))
}

So above we can see she created a new Aspect class that implements the visit method. Each node the aspect visits will be checked to determine if it is an IAM role and if so, set the permission boundary which was passed into the plugin through an environment variable PERMISSION_BOUNDARY_ARN.

Finally for her validate step, she will simply traverse the Terraform config for all IAM roles and check if the permission boundary is set. Even though her preSynth hook will have already done the job, she knows that preSynth is a mutable hook and that another plugin may have altered things after.

// Validate that all IAM roles have a permission boundary
exports.validate = function (config) {
  for (const iamRole of Object.keys(config.resource.aws_iam_role)) {
    const role = config.resource.aws_iam_role[iamRole];
    if (!role.permission_boundary) {
      throw new Error(`Role ${iamRole} does not have a permission boundary`);
    }

    if (role.permission_boundary !== process.env.PERMISSION_BOUNDARY_ARN) {
      throw new Error(`Role ${iamRole} has incorrect permission boundary. Expected: ${process.env.PERMISSION_BOUNDARY_ARN} but got: ${role.permission_boundary}}`);
    }
  }
}

Now Susan can use the plugin in her CD pipelines to ensure that all IAM roles have the correct permission boundary set, without imposing this non-functional requirement on the application developers. Susan will go on to write more plugins to help her organization meet their security and compliance requirements. She is no longer the underappreciated platform engineer we know from the beginning of our blog post, but rather a hero with her own corner office, private parking spot, and an on call pager that never goes off.

Susan Is A Fictional Character

The outcome of her success is purely speculative. Your company may not have corner offices so there is a chance you will have to just settle for the parking spot.

Ask Not What Your Plugin Can Do For You...

This new plugin system is very exciting, and has a lot of possibilities. However, if it is to ever reach its full potential we need your help! If you have some ideas for useful plugins, or thoughts on additional hooks, or even just questions about how to make use of the plugin system, we want to hear from you! Open a pull request or an issue on our GitHub also join our community slack and let us know what you think.

Want to read more about Wing plugins? Check out our plugin documentation for more information on the plugin system. For more code examples visit our plugin code examples