The 8th issue of the Wing Inflight Magazine.

Abstract​

This is an experience report on the initial steps of implementing a CRUD (Create, Read, Update, Delete) REST API in Winglang, with a focus on addressing typical production environment concerns such as secure authentication, observability, and error handling. It highlights how Winglang's distinctive features, particularly the separation of Preflight cloud resource configuration from Inflight API request handling, can facilitate more efficient integration of essential middleware components like logging and error reporting. This balance aims to reduce overall complexity and minimize the resulting code size. The applicability of various design patterns, including Pipe-and-Filters, Decorator, and Factory, is evaluated. Finally, future directions for developing a fully-fledged middleware library for Winglang are identified.

Introduction​

In my previous publication, I reported on my findings about the possible implementation of the Hexagonal Ports and Adapters pattern in the Winglang programming language using the simplest possible GreetingService sample application. The main conclusions from this evaluation were:

1. Cloud resources, such as API Gateway, play the role of drivers (in-) and driven (out-) Ports
2. Event handling functions play the role of Adapters leading to a pure Core, that might be implemented in Winglang, TypeScript, and in fact in any programming language, that compiles into JavaScript and runs on the NodeJS runtime engine

Initially, I planned to proceed with exploring possible ways of implementing a more general Staged Event-Driven Architecture (SEDA) architecture in Winglang. However, using the simplest possible GreetingService as an example left some very important architectural questions unanswered. Therefore I decided to explore in more depth what is involved in implementing a typical Create/Retrieve/Update/Delete (CRUD) service exposing standardized REST API and addressing typical production environment concerns such as secure authentication, observability, error handling, and reporting.

To prevent domain-specific complexity from distorting the focus on important architectural considerations, I chose the simplest possible TODO service with four operations:

1. Retrieve all Tasks (per user)
2. Create a new Task
3. Completely Replace an existing Task definition
4. Delete an existing Task

Using this simple example allowed me to evaluate many important architectural options and to to come up with an initial prototype of a middleware library for the Winglang programming language compatible with and potentially surpassing popular libraries for mainstream programming languages, such as Middy for Node.js middleware engine for AWS Lambda and AWS Power Tools for Lambda.

Unlike my previous publication, I will not describe the step-by-step process of how I arrived at the current arrangement. Software architecture and design processes are rarely linear, especially beyond beginner-level tutorials. Instead, I will describe a starting point solution, which, while far from final, is representative enough to sense the direction in which the final framework might eventually evolve. I will outline the requirements, I wanted to address, the current architectural decisions, and highlight directions for future research.

Simple TODO in Winglang​

Developing a simple, prototype-level TODO REST API service in Winglang is indeed very easy, and could be done within half an hour, using the Winglang Playground:

To keep things simple, I put everything in one source, even though, it of course could be split into Core, Ports, and Adapters. Let’s look at the major parts of this sample.

Resource (Ports) Definition​

First, we need to define cloud resources, aka Ports, that we are going to use. This this is done as follows:

bring ex;
bring cloud;

let tasks = new ex.Table(
  name: "Tasks",
  columns: {
    "id" => ex.ColumnType.STRING,
    "title" => ex.ColumnType.STRING
  },
  primaryKey: "id"
);
let counter = new cloud.Counter();
let api = new cloud.Api();
let path = "/tasks";

Here we define a Winglang Table to keep TODO Tasks with only two columns: task ID and title. To keep things simple, we implement task ID as an auto-incrementing number using the Winglang Counter resource. And finally, we expose the TODO Service API using the Winglang Api resource.

API Request Handlers (Adapters)​

Now, we are going to define a separate handler function for each of the four REST API requests. Getting a list of all tasks is implemented as:

api.get(
  path,
  inflight (request: cloud.ApiRequest): cloud.ApiResponse => {
    let rows = tasks.list();
    let var result = MutArray<Json>[];
    for row in rows {
      result.push(row);
    }
    return cloud.ApiResponse{
      status: 200,
      headers: {
        "Content-Type" => "application/json"
      },
      body: Json.stringify(result)
    };
  });

Creating a new task record is implemented as:

api.post(
  path,
  inflight (request: cloud.ApiRequest): cloud.ApiResponse => {
    let id = "{counter.inc()}";
    if let task = Json.tryParse(request.body) {
      let record = Json{
        id: id,
        title: task.get("title").asStr()
      };
      tasks.insert(id, record);
      return cloud.ApiResponse {
        status: 200,
        headers: {
          "Content-Type" => "application/json"
        },
        body: Json.stringify(record)
      };
    } else {
      return cloud.ApiResponse {
        status: 400,
        headers: {
          "Content-Type" => "text/plain"
        },
        body: "Bad Request"
      };
    }
  });

Updating an existing task is implemented as:

api.put(
  "{path}/:id",
  inflight (request: cloud.ApiRequest): cloud.ApiResponse => {
    let id = request.vars.get("id");
    if let task = Json.tryParse(request.body) {
      let record = Json{
        id: id,
        title: task.get("title").asStr()
      };
      tasks.update(id, record);
      return cloud.ApiResponse {
        status: 200,
        headers: {
          "Content-Type" => "application/json"
        },
        body: Json.stringify(record)
      };
    } else {
      return cloud.ApiResponse {
        status: 400,
        headers: {
          "Content-Type" => "text/plain"
        },
        body: "Bad Request"
      };
    }
  });

Finally, deleting an existing task is implemented as:

api.delete(
  "{path}/:id",
  inflight (request: cloud.ApiRequest): cloud.ApiResponse => {
    let id = request.vars.get("id");
    tasks.delete(id);
    return cloud.ApiResponse {
      status: 200,
      headers: {
        "Content-Type" => "text/plain"
      },
      body: ""
    };
  });

We could play with this API using the Winglang Simulator:

We could write one or more tests to validate the API automatically:

bring http;
bring expect;
let url = "{api.url}{path}";
test "run simple crud scenario" {
  let r1 = http.get(url);
  expect.equal(r1.status, 200);
  let r1_tasks = Json.parse(r1.body);
  expect.nil(r1_tasks.tryGetAt(0));
  let r2 = http.post(url, body: Json.stringify(Json{title: "First Task"}));
  expect.equal(r2.status, 200);
  let r2_task = Json.parse(r2.body);
  expect.equal(r2_task.get("title").asStr(), "First Task");
  let id = r2_task.get("id").asStr();
  let r3 = http.put("{url}/{id}", body: Json.stringify(Json{title: "First Task Updated"}));
  expect.equal(r3.status, 200);
  let r3_task = Json.parse(r3.body);
  expect.equal(r3_task.get("title").asStr(), "First Task Updated");
  let r4 = http.delete("{url}/{id}");
  expect.equal(r4.status, 200);
}

Last but not least, this service can be deployed on any supported cloud platform using the Winglang CLI. The code for the TODO Service is completely cloud-neutral, ensuring compatibility across different platforms without modification.

Should there be a need to expand the task details or link them to other system entities, the approach remains largely unaffected, provided the operations adhere to straightforward CRUD logic and can be executed within a 29-second timeout limit.

This example unequivocally demonstrates that the Winglang programming environment is a top-notch tool for the rapid development of such services. If this is all you need, you need not read further. What follows is a kind of White Rabbit hole of multiple non-functional concerns that need to be addressed before we can even start talking about serious production deployment.

You are warned. The forthcoming text is not for everybody, but rather for seasoned cloud software architects.

Usability​

TODO sample service implementation presented above belongs to the so-called Headless REST API. This approach focuses on core functionality, leaving user experience design to separate layers. This is often implemented as Client-Side Rendering or Server Side Rendering with an intermediate Backend for Frontend tier, or by using multiple narrow-focused REST API services functioning as GraphQL Resolvers. Each approach has its merits for specific contexts.

I advocate for supporting HTTP Content Negotiation and providing a minimal UI for direct API interaction via a browser. While tools like Postman or Swagger can facilitate API interaction, experiencing the API as an end user offers invaluable insights. This basic UI, or what I refer to as an "engineering UI," often suffices.

In this context, anything beyond simple Server Side Rendering deployed alongside headless protocol serialization, such as JSON, might be unnecessarily complex. While Winglang provides support for Website cloud resource for web client assets (HTML pages, JavaScript, CSS), utilizing it for such purposes introduces additional complexity and cost.

A simpler solution would involve basic HTML templates, enhanced with HTMX's features and a CSS framework like Bootstrap. Currently, Winglang does not natively support HTML templates, but for basic use cases, this can be easily managed with TypeScript. For instance, rendering a single task line could be implemented as follows:

import { TaskData } from "core/task";

export function formatTask(path: string, task: TaskData): string {
  return `
    <li class="list-group-item d-flex justify-content-between align-items-center">
        <form hx-put="${path}/${task.taskID}" hx-headers='{"Accept": "text/plain"}' id="${task.taskID}-form">
            <span class="task-text">${task.title}</span>
            <input 
                type="text"
                name="title"
                class="form-control edit-input" 
                style="display: none;" 
                value="${task.title}">
        </form>
        <div class="btn-group">
            <button class="btn btn-danger btn-sm delete-btn" 
                hx-delete="${path}/${task.taskID}" 
                hx-target="closest li" 
                hx-swap="outerHTML" 
                hx-headers='{"Accept": "text/plain"}'>✕</button>
            <button class="btn btn-primary btn-sm edit-btn">✎</button>
        </div>
    </li>
    `;
}

That would result in the following UI screen:

Not super-fancy, but good enough for demo purposes.

Even purely Headless REST APIs require strong usability considerations. API calls should follow REST conventions for HTTP methods, URL formats, and payloads. Proper documentation of HTTP methods and potential error handling are crucial. Client and server errors need to be logged, converted into appropriate HTTP status codes, and accompanied by clear explanation messages in the response body.

The need to handle multiple request parsers and response formatters based on content negotiation using Content-Type and Accept headers in HTTP requests led me to the following design approach:

Adhering to the Dependency Inversion Principle ensures that the system Core is completely isolated from Ports and Adapters. While there might be an inclination to encapsulate the Core within a generic CRUD framework, defined by a ResourceData type, I advise caution. This recommendation stems from several considerations:

1. In practice, even CRUD request processing often entails complexities that extend beyond basic operations.
2. The Core should not rely on any specific framework, preserving its independence and adaptability.
3. The creation of such a framework would necessitate support for Generic Programming, a feature not currently supported by Winglang.

Another option would be to abandon the Core data types definition and rely entirely on untyped JSON interfaces, akin to a Lisp-like programming style. However, given Winglang's strong typing, I decided against this approach.

Overall, the TodoServiceHandler is quite simple and easy to understand:

bring "./data.w" as data;
bring "./parser.w" as parser;
bring "./formatter.w" as formatter;

pub class TodoHandler {
    _path: str;
    _parser: parser.TodoParser;
    _tasks: data.ITaskDataRepository;
    _formatter: formatter.ITodoFormatter;

    new(
        path: str,
        tasks_: data.ITaskDataRepository,
        parser: parser.TodoParser,
        formatter: formatter.ITodoFormatter,
    ) {
        this._path = path;
        this._tasks = tasks_;
        this._parser = parser;
        this._formatter = formatter;
    }

    pub inflight getHomePage(user: Json, outFormat: str): str {
        let userData = this._parser.parseUserData(user);

        return this._formatter.formatHomePage(outFormat, this._path, userData);
    }

    pub inflight getAllTasks(user: Json, query: Map<str>, outFormat: str): str {
        let userData = this._parser.parseUserData(user);
        let tasks = this._tasks.getTasks(userData.userID);

        return this._formatter.formatTasks(outFormat, this._path, tasks);
    }

    pub inflight createTask(
                user: Json,
                body: str,
                inFormat: str,
                outFormat: str
        ): str {
        let taskData = this._parser.parsePartialTaskData(user, body);
        this._tasks.addTask(taskData);

        return this._formatter.formatTasks(outFormat, this._path, [taskData]);
    }

    pub inflight replaceTask(
                user: Json,
                id: str,
                body: str,
                inFormat: str,
                outFormat: str
        ): str {
        let taskData = this._parser.parseFullTaskData(user, id, body);
        this._tasks.replaceTask(taskData);

        return taskData.title;
    }

    pub inflight deleteTask(user: Json, id: str): str {
        let userData = this._parser.parseUserData(user);
        this._tasks.deleteTask(userData.userID, num.fromStr(id));
        return "";
    }
}

As you might notice, the code structure deviates slightly from the design diagram presented earlier. These minor adaptations are normal in software design; new insights emerge throughout the process, necessitating adjustments. The most notable difference is the user: Json argument defined for every function. We'll discuss the purpose of this argument in the next section.

Security​

Exposing the TODO service to the internet without security measures is a recipe for disaster. Hackers, bored teens, and professional attackers will quickly target its public IP address. The rule is very simple:

any public interface must be protected unless exposed for a very short testing period. Security is non-negotiable.

Conversely, overloading a service with every conceivable security measure can lead to prohibitively high operational costs. As I've argued in previous writings, making architects accountable for the costs of their designs might significantly reshape their approach:

If cloud solution architects were responsible for the costs incurred by their systems, it could fundamentally change their design philosophy.

What we need, is a reasonable protection of the service API, not less but not more either. Since I wanted to experiment with full-stack Service Side Rendering UI my natural choice was to enforce user login at the beginning, to produce a JWT Token with reasonable expiration, say one hour, and then to use it for authentication of all forthcoming HTTP requests.

Due to the Service Side Rendering rendering specifics using HTTP Cookie to carry over the session token was a natural (to be honest suggested by ChatGPT) choice. For the Client-Side Rendering option, I might need to use the Bearer Token delivered via the HTTP Request headers Authorization field.

With session tokens now incorporating user information, I could aggregate TODO tasks by the user. Although there are numerous methods to integrate session data, including user details into the domain, I chose to focus on userID and fullName attributes for this study.

For user authentication, several options are available, especially within the AWS ecosystem:

1. AWS Cognito, utilizing its User Pools or integration with external Identity Providers like Google or Facebook.
2. Third-party authentication services such as Auth0.
3. A custom authentication service fully developed in Winglang.
4. AWS Identity Center
5. …

As an independent software technology researcher, I gravitate towards the simplest solutions with the fewest components, which also address daily operational needs. Leveraging the AWS Identity Center, as detailed in a separate publication, was a logical step due to my existing multi-account/multi-user setup.

After integration, my AWS Identity Center main screen looks like this:

That means that in my system, users, myself, or guests, could use the same AWS credentials for development, administration, and sample or housekeeping applications.

To integrate with AWS Identity Center I needed to register my application and provide a new endpoint implementing the so-called “Assertion Consumer Service URL (ACS URL)”. This publication is not about the SAML standard. It would suffice to say that with ChatGPT and Google search assistance, it could be done. Some useful information can be found here. What came very handy was a TypeScript samlify library which encapsulates the whole heavy lifting of the SAML Login Response validation process.

What I’m mostly interested in is how this variability point affects the overall system design. Let’s try to visualize it using a semi-formal data flow notation:

While it might seem unusual this representation reflects with high fidelity how data flows through the system. What we see here is a special instance of the famous Pipe-and-Filters architectural pattern.

Here, data flows through a pipeline and each filter performs one well-defined task in fact following the Single Responsibility Principle. Such an arrangement allows me to replace filters should I want to switch to a simple Basic HTTP Authentication, to use the HTTP Authorization header, or use a different secret management policy for JWT token building and validation.

If we zoom into Parse and Format filters, we will see a typical dispatch logic using Content-Type and Accept HTTP headers respectively:

Many engineers confuse design and architectural patterns with specific implementations. This misses the essence of what patterns are meant to achieve.

Patterns are about identifying a suitable approach to balance conflicting forces with minimal intervention. In the context of building cloud-based software systems, where security is paramount but should not be overpriced in terms of cost or complexity, this understanding is crucial. The Pipe-and-Filters design pattern helps with addressing such design challenges effectively. It allows for modularization and flexible configuration of processing steps, which in this case, relate to authentication mechanisms.

For instance, while robust security measures like SAML authentication are necessary for production environments, they may introduce unnecessary complexity and overhead in scenarios such as automated end-to-end testing. Here, simpler methods like Basic HTTP Authentication may suffice, providing a quick and cost-effective solution without compromising the system's overall integrity. The goal is to maintain the system's core functionality and code base uniformity while varying the authentication strategy based on the environment or specific requirements.

Winglang's unique Preflight compilation feature facilitates this by allowing for configuration adjustments at the build stage, eliminating runtime overhead. This capability presents a significant advantage of Winglang-based solutions over other middleware libraries, such as Middy and AWS Power Tools for Lambda, by offering a more efficient and flexible approach to managing the authentication pipeline.

Implementing Basic HTTP Authentication, therefore, only requires modifying a single filter within the authentication pipeline, leaving the remainder of the system unchanged:

Due to some technical limitations, it’s currently not possible to implement Pipe-and-Filters in Winglang directly, but it could be quite easily simulated by a combination of Decorator and Factory design patterns. How exactly, we will see shortly. Now, let’s proceed to the next topic.

Operation​

In this publication, I’m not going to cover all aspects of production operation. The topic is large and deserves a separate publication of its own. Below, is presented what I consider as a bare minimum:

To operate a service we need to know what happens with it, especially when something goes wrong. This is achieved via a Structured Logging mechanism. At the moment, Winglang provides only a basic log(str) function. For my investigation, I need more and implemented a poor man-structured logging class

// A poor man implementation of configurable Logger
// Similar to that of Python and TypeScript
bring cloud;
bring "./dateTime.w" as dateTime;

pub enum logging {
  TRACE,
  DEBUG,
  INFO,
  WARNING,
  ERROR,
  FATAL
}

//This is just enough configuration
//A serious review including compliance
//with OpenTelemetry and privacy regulations
//Is required. The main insight:
//Serverless Cloud logging is substantially
//different
pub interface ILoggingStrategy {
  inflight timestamp(): str;
  inflight print(message: Json): void;
}

pub class DefaultLoggerStrategy impl ILoggingStrategy {
  pub inflight timestamp(): str {
    return dateTime.DateTime.toUtcString(std.Datetime.utcNow());
  }
  pub inflight print(message: Json): void {
    log("{message}");
  }
}

//TBD: probably should go into a separate module
bring expect;
bring ex;

pub class MockLoggerStrategy impl ILoggingStrategy {
  _name: str;
  _counter: cloud.Counter;
  _messages: ex.Table;

  new(name: str?) {
    this._name = name ?? "MockLogger";
    this._counter = new cloud.Counter();
    this._messages = new ex.Table(
      name: "{this._name}Messages",
      columns: Map<ex.ColumnType>{
        "id" => ex.ColumnType.STRING,
        "message" => ex.ColumnType.STRING
      },
      primaryKey: "id"
    );
  }
  pub inflight timestamp(): str {
    return "{this._counter.inc(1, this._name)}";
  }
  pub inflight expect(messages: Array<Json>): void {
    for message in messages {
      this._messages.insert(
          message.get("timestamp").asStr(),
          Json{ message: "{message}"}
        );
    }
  }
  pub inflight print(message: Json): void {
    let expected = this._messages.get(
        message.get("timestamp").asStr()
      ).get("message").asStr();
    expect.equal("{message}", expected);
  }
}

pub class Logger {
  _labels: Array<str>;
  _levels: Array<logging>;
  _level: num;
  _service: str;
  _strategy: ILoggingStrategy;

  new (level: logging, service: str, strategy: ILoggingStrategy?) {
    this._labels = [
        "TRACE",
        "DEBUG",
        "INFO",
        "WARNING",
        "ERROR",
        "FATAL"
    ];
    this._levels = Array<logging>[
      logging.TRACE,
      logging.DEBUG,
      logging.INFO,
      logging.WARNING,
      logging.ERROR,
      logging.FATAL
    ];
    this._level = this._levels.indexOf(level);
    this._service = service;
    this._strategy = strategy ?? new DefaultLoggerStrategy();
  }
  pub inflight log(level_: logging, func: str, message: Json): void {
    let level = this._levels.indexOf(level_);
    let label = this._labels.at(level);
    if  this._level <= level {
      this._strategy.print(Json {
        timestamp: this._strategy.timestamp(),
        level: label,
        service: this._service,
        function: func,
        message: message
      });
    }
  }
  pub inflight trace(func: str, message: Json): void {
    this.log(logging.TRACE, func,message);
  }
  pub inflight debug(func: str, message: Json): void {
    this.log(logging.DEBUG, func, message);
  }
  pub inflight info(func: str, message: Json): void {
    this.log(logging.INFO, func, message);
  }
  pub inflight warning(func: str, message: Json): void {
    this.log(logging.WARNING, func, message);
  }
  pub inflight error(func: str, message: Json): void {
    this.log(logging.ERROR, func, message);
  }
  pub inflight fatal(func: str, message: Json): void {
    this.log(logging.FATAL, func, message);
  }
}

There is nothing spectacular here and, as I wrote in the comments, a cloud-based logging system requires a serious revision. Still, it’s enough for the current investigation. I’m fully convinced that logging is an integral part of any service specification and has to be tested with the same rigor as core functionality. For that purpose, I developed a simple mechanism to mock logs and check them against expectations.

For a REST API CRUD service, we need to log at least three types of things:

1. HTTP Request
2. Original Error message if something wrong happened
3. HTTP Response

In addition, depending on needs the original error message might need to be converted into a standard one, for example in order not to educate attackers.

How much if any details to log depends on multiple factors: deployment target, type of request, specific user, type of error, statistical sampling, etc. In development and test mode, we will normally opt for logging almost everything and returning the original error message directly to the client screen to ease debugging. In production mode, we might opt for removing some sensitive data because of regulation requirements, to return a general error message, such as “Bad Request”, without any details, and apply only statistical sample logging for particular types of requests to save the cost.

Flexible logging configuration was achieved by injecting four additional filters in every request handling pipeline:

1. HTTP Request logging filter
2. Try/Catch Decorator to convert exceptions if any into HTTP status codes and to log original error messages (this could be extracted into a separate filter, but I decided to keep things simple)
3. Error message translator to convert original error messages into standard ones if required
4. HTTP Response logging filter

This structure, although not an ultimate one, provides enough flexibility to implement a wide range of logging and error-handling strategies depending on the service and its deployment target specifics.

As with logs, Winglang at the moment provides only a basic throw <str> operator, so I decided to implement my version of a poor man structured exceptions:

// A poor man structured exceptions
pub inflight class Exception {
    pub tag: str;
    pub message: str?;

    new(tag: str, message: str?) {
        this.tag = tag;
        this.message = message;
    }
    pub raise() {
        let err = Json.stringify(this);
        throw err;
    }
    pub static fromJson(err: str): Exception {
        let je = Json.parse(err);

        return new Exception(
            je.get("tag").asStr(),
            je.tryGet("message")?.tryAsStr()
          );
    }
    pub toJson(): Json { //for logging
        return Json{tag: this.tag, message: this.message};
    }
}

// Standard exceptions, similar to those of Python
pub inflight class KeyError extends Exception {
    new(message: str?) {
        super("KeyError", message);
    }
}
pub inflight class ValueError extends Exception {
    new(message: str?) {
        super("ValueError", message);
    }
}
pub inflight class InternalError extends Exception {
    new(message: str?) {
        super("InternalError", message);
    }
}
pub inflight class NotImplementedError extends Exception {
    new(message: str?) {
        super("NotImplementedError", message);
    }
}
//Two more HTTP-specific, yet useful
pub inflight class AuthenticationError extends Exception {
        //aka HTTP 401 Unauthorized
    new(message: str?) {
        super("AuthenticationError", message);
    }
}
pub inflight class AuthorizationError extends Exception {
        //aka HTTP 403 Forbidden
    new(message: str?) {
        super("AuthorizationError", message);
    }
}

These experiences highlight how the developer community can bridge gaps in new languages with temporary workarounds. Winglang is still evolving, but its innovative features can be harnessed for progress despite the language's age.

Now, it’s time to take a brief look at the last production topic on my list, namely

Scale​

Scaling is a crucial aspect of cloud development, but it's often misunderstood. Some neglect it entirely, leading to problems when the system grows. Others over-engineer, aiming to be a "FANG" system from day one. The proclamation "We run everything on Kubernetes" is a common refrain in technical circles, regardless of whether it's appropriate for the project at hand.

Neither—neglect nor over-engineering— extreme is ideal. Like security, scaling shouldn't be ignored, but it also shouldn't be over-emphasized.

Up to a certain point, cloud platforms provide cost-effective scaling mechanisms. Often, the choice between different options boils down to personal preference or inertia rather than significant technical advantages.

The prudent path involves starting small and cost-effectively, scaling out based on real-world usage and performance data, rather than assumptions. This approach necessitates a system designed for easy configuration changes to accommodate scaling, something not inherently supported by Winglang but certainly within the realm of feasibility through further development and research. As an illustration, let's consider scaling within the AWS ecosystem:

1. Initially, a cost-effective and swift deployment might involve a single Lambda Function URL for a full-stack CRUD API with Server-Side Rendering, using an S3 Bucket for storage. This setup enables rapid feedback essential for early development stages. Personally, I favor a "UX First" approach over "API First." You might be surprised how far you can get with this basic technology stack. While Winglang doesn't currently support Lambda Function URLs, I believe it could be achieved with filter combinations and system adjustments. At this level, following Marc Van Neerven's recommendation to use standard Web Components instead of heavy frameworks could be beneficial. This is a subject for future exploration.
2. Transitioning to an API Gateway or GraphQL Gateway becomes relevant when external API exposure or advanced features like WebSockets are required. If the initial data storage solution becomes a bottleneck, it might be time to consider switching to a more robust and scalable solution like DynamoDB. At this point, deploying separate Lambda Functions for each API request might offer simplicity in implementation, though it's not always the most cost-effective strategy.
3. The move to containerized solutions should be data-driven, considered only when there's clear evidence that the function-based architecture is either too costly or suffers from latency issues due to cold starts. An initial foray into containers might involve using ECS Fargate for its simplicity and cost-effectiveness, reserving EKS for scenarios with specific operational needs that require its capabilities. This evolution should ideally be managed through configuration adjustments and automated filter generation, leveraging Winglang's unique capabilities to support dynamic scaling strategies.

In essence, Winglang's approach, emphasizing the Preflight and Inflight stages, holds promise for facilitating these scaling strategies, although it may still be in the early stages of fully realizing this potential. This exploration of scalability within cloud software development emphasizes starting small, basing decisions on actual data, and remaining flexible in adapting to changing requirements.

Concluding Remarks

In the mid-1990s, I learned about Commonality Variability Analysis from Jim Coplien. Since then, this approach, alongside Edsger W. Dijkstra's Layered Architecture, has been a cornerstone of my software engineering practices. Commonality Variability Analysis asks: "In our system, which parts will always be the same and which might need to change?" The Open-Closed Principle dictates that variable parts should be replaceable without modifying the core system.

Deciding when to finalize the stable aspects of a system involves navigating the trade-off between flexibility and efficiency, with several stages from code generation to runtime offering opportunities for fixation. Dynamic language proponents might delay these decisions to runtime for maximum flexibility, whereas advocates for static, compiled languages typically secure crucial system components as early as possible.

Winglang, with its unique Preflight compilation phase, stands out by allowing cloud resources to be fixed early in the development process. In this publication, I explored how Winglang enables addressing non-functional aspects of cloud services through a flexible pipeline of filters, though this granularity introduces its own complexity. The challenge now becomes managing this complexity without compromising the system's efficiency or flexibility.

While the final solution is a work in progress, I can outline a high-level design that balances these forces:

This design combines several software Design Patterns to achieve the desired balance. The process involves:

1. The Pipeline Builder component is responsible for preparing a final set of Preflight components.
2. The Pipeline Builder reads a Configuration which might be organized as a Composite (think team-wide or organization-wide configuration).
3. Configurations specify capability requirements for resources (e.g., loggers).
4. Each Resource has several Specifications each one defining conditions under which a Factory needs to be invoked to produce the required Filter. Three filter types are envisioned:
   1. Row HTTP Request/Response Filter
   2. Extended HTTP Request/Response Filter with session information extracted after token validation
   3. Generic CRUD requests filter to be forwarded to Core

This approach shifts complexity towards implementing the Pipeline Builder machinery and Configuration specification. Experience teaches such machinery could be implemented (described for example in this publication). That normally requires some generic programming and dynamic import capabilities. Coming up with a good configuration data model is more challenging.

Recent advances in generative AI-based copilots raise new questions about achieving the most cost-efficient outcome. To understand the problem, let's revisit the traditional compilation and configuration stack:

This general case may not apply to every ecosystem. Here's a breakdown of the typical layers:

1. The Core Language is designed to be small (”C” and Lisp tradition). It may or may not provide support for Reflection.
2. As many as possible extended capabilities are provided by Standard Library and 3rd Party Libraries and Frameworks.
3. Generic Meta-Programming: Support for features like C++ templates or Lisp macros is introduced early (C++, Rust) or later (Java, C#). Generics are a source of ongoing debate:
   1. Framework developers find them insufficiently expressive.
   2. Application developers struggle with their complexity.
   3. Scala exemplifies the potential downsides of overly complex generics.
4. Despite criticism, macros (e.g., C preprocessor) persist as a tool for automated code generation, often compensating for generic limitations.
5. Third-party vendors (often open-source) provide solutions that enhance or compensate for the standard library, typically using external configuration files (YAML, JSON, etc.).
6. Specialized generators very often use external blueprints or templates.

This complex structure has limitations. Generics can obscure the core language, macros are unsafe, configuration files are poorly disguised scripts, and code generators rely on inflexible static templates. These limitations are why I believe the current trend of Internal Development Platforms has limited growth potential.

As we look forward to the role of generative AI in streamlining these processes, the question becomes: Can generative AI-based copilots not only simplify but also enhance our ability to balance commonality and variability in software engineering?

This is going to be the main topic of my future research to be reported in the next publications. Stay tuned.

The landscape of serverless architectures is constantly evolving - and the patterns used by developers today range from microservices, to hexagonal architectures, to multi-tier architectures, to event-based architectures (EDAs). One of the most important challenges every development team considers when designing and building their system is how well it secures their application.

In this post, we’re going to deep dive on the private API gateway pattern: including what a private API gateway entails, its distinct advantages, and why you or your team might opt for this route over other services.

Based on our team’s experience, we’ll be focusing on the capabilities available when building private API gateways on AWS using their managed VPC and API Gateway services. But many of the lessons will also apply to cloud applications built using Azure’s API Management service and other major cloud providers.

Understanding Private API Gateways​

If we simplify what a private API gateway is, it's a secure means of exposing a set of APIs within a private network, typically established using a Virtual Private Cloud (VPC). Let’s first understand each of these.

An API Gateway makes it easy for developers to create, publish, maintain, monitor, and secure large numbers of API endpoints at scale. The gateway provides a central point where you can manage API throttling, authorization, API versioning, and monitoring configuration.

On the other hand, a Virtual Private Cloud (or VPC), is a mechanism for creating a logically isolated virtual networking environment. Such a virtual network parallels a network that you’d operate in your own data center, including subnets, IP addressing capabilities, routing tables, and gateways to connect to other networks.

When your API gateway is located within a VPC, you have a private API gateway. Unlike their public counterparts accessible over the internet, private API gateways are crafted to be accessed exclusively from within the specified network. This means only backend services and databases created within your organization can access the API endpoints.

Why Choose a Private API Gateway?​

Let's take a look at three common use cases.

Improved security:​

By confining your API within a private network, you minimize exposure to potential security threats originating from a public API over the internet. For the security conscious, VPCs provide a much saner starting point for securing your application since private compute resources can simply say “allow any traffic from our private network,” reducing the need to manually manage firewalls and IP tables. In this scenario, it doesn’t really matter what you’re building - you just don’t want the public internet messing around with it!

Compliance:​

Some compliance standards or organizational policies mandate the use of VPCs to ensure that sensitive data remains within controlled environments. For example, with regulations like HIPAA in the United States, deploying applications within VPCs is a common strategy to ensure the confidentiality and security of patient data. In these situations, cloud architectures that are designed around API endpoint usage will benefit from being able to use private API gateways.

Hybrid cloud environments:​

For companies that operate in a hybrid cloud environment (a mix of public cloud and on-premises data centers), a private API gateway can manage and route traffic within the private network. This is essential for sensitive data that cannot be transferred to public clouds due to policy or regulatory reasons. Migrating applications to the cloud is made easier by the fact that modern cloud providers like AWS allow application code running within VPCs to have secure access to the common services like S3, DynamoDB, and IAM - all routed through the backbone of Amazon’s networking infrastructure.

Challenges​

While setting up cloud applications in private networks has many security benefits, accessing a private API from external environments, such as during development or testing phases, can be cumbersome. VPCs typically add complexity to serverless applications by increasing the amount of infrastructure that needs to be managed through tools like Terraform and CloudFormation, and may require setting up bastion hosts for debugging applications in production.

Furthermore, establishing secure connections to any networks outside of the VPC may require setting up Virtual Private Networks (VPNs) or using services like Amazon Direct Connect, which introduces extra complexity.

Overcoming it​

In the Wing Slack community, we’ve seen many developers share their advice and experiences building cloud applications of all shapes and sizes. Recently, we heard about a user’s positive experience building a cloud application with Wing that established API endpoints within their VPC through a private API Gateway. Inspired by their efforts, we extracted their solution into a template that makes it easy for anyone to turn their API gateway into a private one.

One of the biggest pains people have when using serverless is that they constantly have to wait for deployments to finish in order to test changes to their code. This is where Winglang and it's cloud simulator rocks - it allows you to develop your entire serverless application without having to deploy anything on the cloud, letting you iterate in milliseconds instead of minutes.

If you’re curious to learn more, check out our tutorial that walks you step-by-step through building a simple application with a private API gateway and deploy it to your own AWS account. If you have feedback or any other questions, let us know by dropping a comment in our community slack. Don’t be shy!

TL;DR​

As the saying goes, there are several ways to skin a cat...in the tech world, there are 5 ways to skin a Lambda Function 🤩

Lets Compare 5 DevTools​

• ✅ Wing

• ✅ Pulumi

• ✅ AWS-CDK

• ✅ CDK for Terraform

• ✅ Terraform

Introduction​

As developers try to bridge the gap between development and DevOps, I thought it would be helpful to compare Programming Languages and DevTools.

Let's start with the idea of a simple function that would upload a text file to a Bucket in our cloud app.

The next step is to demonstrate several ways this could be accomplished.

Note: In cloud development, managing permissions and bucket identities, packaging runtime code, and handling multiple files for infrastructure and runtime add layers of complexity to the development process.

Let's dive into some code!

1. Wing​

After installing Wing, let's create a file: main.w

If you aren't familiar with the Wing Programming Language, please check out the open-source repo HERE

bring cloud;

let bucket = new cloud.Bucket();

new cloud.Function(inflight () => {
  bucket.put("hello.txt", "world!");
});

Let's do a breakdown of what's happening in the code above.

bring cloud is Wing's import syntax

Create a Cloud Bucket: let bucket = new cloud.Bucket(); initializes a new cloud bucket instance.

On the backend, the Wing platform provisions a new bucket in your cloud provider's environment. This bucket is used for storing and retrieving data.

Create a Cloud Function: The new cloud.Function(inflight () => { ... }); statement defines a new cloud function.

This function, when triggered, performs the actions defined within its body.

bucket.put("hello.txt", "world!"); uploads a file named hello.txt with the content world! to the cloud bucket created earlier.

Compile & Deploy to AWS​

• wing compile --platform tf-aws main.w

• terraform apply

That's it, Wing takes care of the complexity of (permissions, getting the bucket identity in the runtime code, packaging the runtime code into a bucket, having to write multiple files - for infrastructure and runtime), etc.

Not to mention it generates IAC (TF or CF), plus Javascript that you can deploy with existing tools.

But while you develop, you can use the local simulator to get instant feedback and shorten the iteration cycles

Wing even has a playground that you can try out in the browser!

2. Pulumi​

Step 1: Initialize a New Pulumi Project

mkdir pulumi-s3-lambda-ts
cd pulumi-s3-lambda-ts
pulumi new aws-typescript

Step 2. Write the code to upload a text file to S3.

This will be your project structure.

pulumi-s3-lambda-ts/
├─ src/
│  ├─ index.ts              # Pulumi infrastructure code
│  └─ lambda/
│     └─ index.ts           # Lambda function code to upload a file to S3
├─ tsconfig.json            # TypeScript configuration
└─ package.json             # Node.js project file with dependencies

Let's add this code to index.ts

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an AWS S3 bucket
const bucket = new aws.s3.Bucket("myBucket", {
  acl: "private",
});

// IAM role for the Lambda function
const lambdaRole = new aws.iam.Role("lambdaRole", {
  assumeRolePolicy: JSON.stringify({
    Version: "2023-10-17",
    Statement: [
      {
        Action: "sts:AssumeRole",
        Principal: {
          Service: "lambda.amazonaws.com",
        },
        Effect: "Allow",
        Sid: "",
      },
    ],
  }),
});

// Attach the AWSLambdaBasicExecutionRole policy
new aws.iam.RolePolicyAttachment("lambdaExecutionRole", {
  role: lambdaRole,
  policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});

// Policy to allow Lambda function to access the S3 bucket
const lambdaS3Policy = new aws.iam.Policy("lambdaS3Policy", {
  policy: bucket.arn.apply((arn) =>
    JSON.stringify({
      Version: "2023-10-17",
      Statement: [
        {
          Action: ["s3:PutObject", "s3:GetObject"],
          Resource: `${arn}/*`,
          Effect: "Allow",
        },
      ],
    })
  ),
});

// Attach policy to Lambda role
new aws.iam.RolePolicyAttachment("lambdaS3PolicyAttachment", {
  role: lambdaRole,
  policyArn: lambdaS3Policy.arn,
});

// Lambda function
const lambda = new aws.lambda.Function("myLambda", {
  code: new pulumi.asset.AssetArchive({
    ".": new pulumi.asset.FileArchive("./src/lambda"),
  }),
  runtime: aws.lambda.Runtime.NodeJS12dX,
  role: lambdaRole.arn,
  handler: "index.handler",
  environment: {
    variables: {
      BUCKET_NAME: bucket.bucket,
    },
  },
});

export const bucketName = bucket.id;
export const lambdaArn = lambda.arn;

Next, create a lambda/index.ts directory for the Lambda function code:

import { S3 } from "aws-sdk";

const s3 = new S3();

export const handler = async (): Promise<void> => {
  const bucketName = process.env.BUCKET_NAME || "";
  const fileName = "example.txt";
  const content = "Hello, Pulumi!";

  const params = {
    Bucket: bucketName,
    Key: fileName,
    Body: content,
  };

  try {
    await s3.putObject(params).promise();
    console.log(
      `File uploaded successfully at https://${bucketName}.s3.amazonaws.com/${fileName}`
    );
  } catch (err) {
    console.log(err);
  }
};

Step 3: TypeScript Configuration (tsconfig.json)

{
  "compilerOptions": {
    "target": "ES2018",
    "module": "CommonJS",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules", "**/*.spec.ts"]
}

After creating a Pulumi project, a yaml file will automatically be generated. pulumi.yaml

name: s3-lambda-pulumi
runtime: nodejs
description: A simple example that uploads a file to an S3 bucket using a Lambda function
template:
  config:
    aws:region:
      description: The AWS region to deploy into
      default: us-west-2

Deploy with Pulumi​

Ensure your lambda directory with the index.js file is correctly set up. Then, run the following command to deploy your infrastructure: pulumi up

3. AWS-CDK​

Step 1: Initialize a New CDK Project

mkdir cdk-s3-lambda
cd cdk-s3-lambda
cdk init app --language=typescript

Step 2: Add Dependencies

npm install @aws-cdk/aws-lambda @aws-cdk/aws-s3

Step 3: Define the AWS Resources in CDK

File: index.js

import * as cdk from "@aws-cdk/core";
import * as lambda from "@aws-cdk/aws-lambda";
import * as s3 from "@aws-cdk/aws-s3";

export class CdkS3LambdaStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create the S3 bucket
    const bucket = new s3.Bucket(this, "MyBucket", {
      removalPolicy: cdk.RemovalPolicy.DESTROY, // NOT recommended for production code
    });

    // Define the Lambda function
    const lambdaFunction = new lambda.Function(this, "MyLambda", {
      runtime: lambda.Runtime.NODEJS_14_X, // Define the runtime
      handler: "index.handler", // Specifies the entry point
      code: lambda.Code.fromAsset("lambda"), // Directory containing your Lambda code
      environment: {
        BUCKET_NAME: bucket.bucketName,
      },
    });

    // Grant the Lambda function permissions to write to the S3 bucket
    bucket.grantWrite(lambdaFunction);
  }
}

Step 4: Lambda Function Code

Create the same file struct as above and in the pulumi directory: index.ts

import { S3 } from 'aws-sdk';
const s3 = new S3();

exports.handler = async (event: any) => {
  const bucketName = process.env.BUCKET_NAME;
  const fileName = 'uploaded_file.txt';
  const content = 'Hello, CDK! This file was uploaded by a Lambda function!';

  try {
    const result = await s3.putObject({
      Bucket: bucketName!,
      Key: fileName,
      Body: content,
    }).promise();

    console.log(`File uploaded successfully: ${result}`);
    return {
      statusCode: 200,
      body: `File uploaded successfully: ${fileName}`,
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: 500,
      body: `Failed to upload file: ${error}`,
    };
  }
};

Deploy the CDK Stack​

First, compile your TypeScript code: npm run build, then

Deploy your CDK to AWS: cdk deploy

4. CDK for Terraform​

Step 1: Initialize a New CDKTF Project

mkdir cdktf-s3-lambda-ts
cd cdktf-s3-lambda-ts

Then, initialize a new CDKTF project using TypeScript:

cdktf init --template="typescript" --local

Step 2: Install AWS Provider and Add Dependencies

npm install @cdktf/provider-aws

Step 3: Define the Infrastructure

Edit main.ts to define the S3 bucket and Lambda function:

import { Construct } from "constructs";
import { App, TerraformStack } from "cdktf";
import { AwsProvider, s3, lambdafunction, iam } from "@cdktf/provider-aws";

class MyStack extends TerraformStack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new AwsProvider(this, "aws", { region: "us-west-2" });

    // S3 bucket
    const bucket = new s3.S3Bucket(this, "lambdaBucket", {
      bucketPrefix: "cdktf-lambda-",
    });

    // IAM role for Lambda
    const role = new iam.IamRole(this, "lambdaRole", {
      name: "lambda_execution_role",
      assumeRolePolicy: JSON.stringify({
        Version: "2023-10-17",
        Statement: [
          {
            Action: "sts:AssumeRole",
            Principal: { Service: "lambda.amazonaws.com" },
            Effect: "Allow",
          },
        ],
      }),
    });

    new iam.IamRolePolicyAttachment(this, "lambdaPolicy", {
      role: role.name,
      policyArn:
        "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
    });

    const lambdaFunction = new lambdafunction.LambdaFunction(this, "MyLambda", {
      functionName: "myLambdaFunction",
      handler: "index.handler",
      role: role.arn,
      runtime: "nodejs14.x",
      s3Bucket: bucket.bucket, // Assuming the Lambda code is uploaded to this bucket
      s3Key: "lambda.zip", // Assuming the Lambda code zip file is named lambda.zip
      environment: {
        variables: {
          BUCKET_NAME: bucket.bucket,
        },
      },
    });

    // Grant the Lambda function permissions to write to the S3 bucket
    new s3.S3BucketPolicy(this, "BucketPolicy", {
      bucket: bucket.bucket,
      policy: bucket.bucket.apply((name) =>
        JSON.stringify({
          Version: "2023-10-17",
          Statement: [
            {
              Action: "s3:*",
              Resource: `arn:aws:s3:::${name}/*`,
              Effect: "Allow",
              Principal: {
                AWS: role.arn,
              },
            },
          ],
        })
      ),
    });
  }
}

const app = new App();
new MyStack(app, "cdktf-s3-lambda-ts");
app.synth();

Step 4: Lambda Function Code

The Lambda function code should be written in TypeScript and compiled into JavaScript, as AWS Lambda natively executes JavaScript. Here's an example index.ts for the Lambda function that you need to compile and zip:

import { S3 } from "aws-sdk";

const s3 = new S3();

exports.handler = async () => {
  const bucketName = process.env.BUCKET_NAME || "";
  const content = "Hello, CDKTF!";
  const params = {
    Bucket: bucketName,
    Key: `upload-${Date.now()}.txt`,
    Body: content,
  };

  try {
    await s3.putObject(params).promise();
    return { statusCode: 200, body: "File uploaded successfully" };
  } catch (err) {
    console.error(err);
    return { statusCode: 500, body: "Failed to upload file" };
  }
};

You need to compile this TypeScript code to JavaScript, zip it, and upload it to the S3 bucket manually or using a script.

Ensure the s3Key in the LambdaFunction resource points to the correct zip file in the bucket.

Compile & Deploy Your CDKTF Project​

Compile your project using npm run build

Generate Terraform Configuration Files

Run the cdktf synth command. This command executes your CDKTF app, which generates Terraform configuration files (*.tf.json files) in the cdktf.out directory:

Deploy Your Infrastructure

cdktf deploy

5. Terraform​

Step 1: Terraform Setup

Define your AWS Provider and S3 Bucket Create a file named main.tf with the following:

provider "aws" {
  region = "us-west-2" # Choose your AWS region
}

resource "aws_s3_bucket" "lambda_bucket" {
  bucket_prefix = "lambda-upload-bucket-"
  acl           = "private"
}

resource "aws_iam_role" "lambda_execution_role" {
  name = "lambda_execution_role"

  assume_role_policy = jsonencode({
    Version = "2023-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_policy" "lambda_s3_policy" {
  name        = "lambda_s3_policy"
  description = "IAM policy for Lambda to access S3"

  policy = jsonencode({
    Version = "2023-10-17"
    Statement = [
      {
        Action   = ["s3:PutObject", "s3:GetObject"],
        Effect   = "Allow",
        Resource = "${aws_s3_bucket.lambda_bucket.arn}/*"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_s3_access" {
  role       = aws_iam_role.lambda_execution_role.name
  policy_arn = aws_iam_policy.lambda_s3_policy.arn
}

resource "aws_lambda_function" "uploader_lambda" {
  function_name = "S3Uploader"

  s3_bucket = "YOUR_DEPLOYMENT_BUCKET_NAME" # Set your deployment bucket name here
  s3_key    = "lambda.zip" # Upload your ZIP file to S3 and set its key here

  handler = "index.handler"
  role    = aws_iam_role.lambda_execution_role.arn
  runtime = "nodejs14.x"

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.lambda_bucket.bucket
    }
  }
}

Step 2: Lambda Function Code (TypeScript)

Create a TypeScript file index.ts for the Lambda function:

import { S3 } from 'aws-sdk';

const s3 = new S3();

exports.handler = async (event: any) => {
  const bucketName = process.env.BUCKET_NAME;
  const fileName = `uploaded-${Date.now()}.txt`;
  const content = 'Hello, Terraform and AWS Lambda!';

  try {
    await s3.putObject({
      Bucket: bucketName!,
      Key: fileName,
      Body: content,
    }).promise();

    console.log('Upload successful');
    return {
      statusCode: 200,
      body: JSON.stringify({ message: 'Upload successful' }),
    };
  } catch (error) {
    console.error('Upload failed:', error);
    return {
      statusCode: 500,
      body: JSON.stringify({ message: 'Upload failed' }),
    };
  }
};