🔭Getting Hands-On with Pulumi: A Python Guide to Managing Your AWS Resources

Hello, and welcome to your Aiden's Lab Notebook.

In our last post, we explored why Infrastructure as Code (IaC) can be so valuable and took a look at how Pulumi works.

This time, I wanted to walk you through the core workflow of Pulumi by showing you how I deployed an AWS SQS queue.

I've put this guide together based on my own testing, so if you're curious about IaC and Pulumi, I really encourage you to check it out!

1. Setting Up: Installing Pulumi and Configuring AWS

Before we can start using Pulumi, we first need to install the Pulumi CLI on our machine and get our AWS account configured.

You'll also need to have the Python package manager pip and the virtual environment tool venv installed beforehand.

Since I'm using a WSL environment, I installed Pulumi by running the following command in my terminal:

curl -fsSL https://get.pulumi.com | sh

The installation script runs automatically, and once it's done, you should be able to check your Pulumi version like this:

To use Pulumi, you need to either log in to the Pulumi Cloud or log in locally. For this quick hands-on session, we'll log in locally using the command below:

pulumi login --local

You should then see a message indicating you're logged in with your current username.

Next up is configuring your cloud account. The Pulumi CLI needs to know your AWS account details to create or manage AWS resources.

If you already have the AWS CLI installed and configured locally, you might not need to do any extra setup. However, if you haven't installed the AWS CLI, or if you're planning to use Pulumi in a CI/CD environment, you'll need to have your AWS Access Key ID and AWS Secret Access Key stored as local environment variables, like so:

export AWS_ACCESS_KEY_ID="<YOUR_AWS_ACCOUNT_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<YOUR_AWS_ACCOUNT_SECRET_ACCESS_KEY>"

2. Creating a New Pulumi Project

Alright, we're all set to begin with Pulumi! Navigate to your desired folder and let's create a new Pulumi project.

Since this guide focuses on managing AWS infrastructure with Python, we'll use the aws-python template by running this command:

pulumi new aws-python

As you can see in the image below, the terminal will then prompt you to set the project name, description, stack name, a passphrase for encrypting secrets, your preferred package manager, and the AWS region.

If you've made it this far without a hitch, then congratulations, your Pulumi project has been successfully created! 🎉

You should see that a basic project structure and some example code have been generated in your directory, like this:

Here are the core files essential for Pulumi to work:

• Pulumi.yaml: Contains metadata about your project.

• Pulumi.<stack-name>.yaml: Stores configuration values for a specific stack (e.g., dev, prod).

• __main__.py: This is where you'll write your actual infrastructure code.

• requirements.txt: Manages your project's dependencies.

You might have been wondering about the term 'stack,' as it's come up a few times.

Pulumi allows you to use the same infrastructure code to configure multiple, isolated instances of that infrastructure. In Pulumi's world, these are called stacks.

For example, you could manage different environments like dev, staging, and production all using the same codebase.

Each stack has its own configuration settings and state information, so changes to one stack don't affect the others.

3. Writing Your Pulumi Infrastructure Code

Let's get back to our Pulumi project. When you created the new project, the actual infrastructure code file (__main__.py) was generated with some example code for creating an AWS S3 bucket.

While the S3 bucket example is good, for this guide, I've prepared an example that creates an AWS SQS resource, which is often used to enhance application scalability and reliability.

For those curious about what AWS SQS is:

SQS is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. It lets you send, store, and receive messages between software components reliably.

Now, let's add the following AWS SQS creation example code to our infrastructure code file (__main__.py):

import pulumi
import pulumi_aws as aws

# Create an SQS queue
# "mySimpleQueue" is the logical name Pulumi uses to identify this resource within the program.
my_queue = aws.sqs.Queue("mySimpleQueue",
    name="my-app-queue",  # The actual name of the queue to be created (optional; Pulumi auto-generates if not specified)
    tags={
        "Environment": "dev",
        "Project": "MyNewsletterApp",
    })

# Export information about the created SQS Queue
# Using pulumi.export() will print the value to the terminal after pulumi up runs.
pulumi.export("queue_url", my_queue.id) # Returns the URL of the created SQS queue
pulumi.export("queue_arn", my_queue.arn) # Returns the Amazon Resource Name (ARN) of the created SQS queue

4. Deploying Infrastructure Resources with Pulumi

Once you've finished writing your infrastructure code, the next step is to deploy the actual infrastructure resources using Pulumi, right?

Of course, you could deploy immediately, but to manage your infrastructure reliably, I recommend following this workflow:

1. Run the pulumi preview command to see a preview of the planned changes without actually altering your infrastructure.

2. Run the pulumi up command to create and update your actual infrastructure.

When you run pulumi preview first, it will prompt you for your passphrase (if you set one) and then show you a preview, like this:

You can see that the SQS queue we intended to create in our infrastructure code is nicely displayed in the preview.

Now, let's deploy the actual AWS SQS queue using the pulumi up command!

When you run this command, just like with pulumi preview, it will first ask for your passphrase and then show you a preview of the changes.

Then, when it asks if you want to perform the update, select "yes." As you can see below, the actual infrastructure resources will be created, and then the SQS URL and ARN will be displayed.

Here’s the SQS page in the AWS console before running the pulumi up command. As you can see, there are no queues created yet.

However, once the pulumi up command completes, you'll see a new queue with the name we defined in our infrastructure code (my-app-queue) appear on the AWS console page, just like this:

If you click on the queue name to see its details, you'll find that the tags we defined in our infrastructure code ("Environment": "dev", "Project": "MyNewsletterApp") have been correctly applied.

5. Removing Infrastructure Resources Deployed with Pulumi

So far, we've walked through writing and deploying infrastructure code.

Now, let's also look at how to remove the resources we've deployed.

Removing infrastructure resources is just as straightforward as deploying them.

In the terminal, from the same project directory, you just need to run the pulumi destroy command and then enter your passphrase, like so:

The pulumi destroy command safely removes the infrastructure resources managed by that stack, but the stack's configuration and history still remain.

This information, as we discussed earlier, is stored in the state management system.

If you want to remove all information related to the stack as well, you can use the pulumi stack rm command.

However, there are two important things to keep in mind, so please make sure to note them:

1. Before running pulumi stack rm, you must first run pulumi destroy to remove all infrastructure resources managed by that stack.

2. Be very careful, as once stack information is removed, it's difficult to recover.

When you run pulumi stack rm in the terminal, you'll need to type the name of the stack you want to remove (e.g., dev) to confirm the deletion.

So, we've walked through managing AWS resources with Pulumi by following this core workflow:

1. Install Pulumi and Configure Your Account

2. Create a New Pulumi Project

3. Write Pulumi Infrastructure Management Code

4. Deploy Infrastructure Resources with Pulumi

5. Remove Infrastructure Resources Deployed with Pulumi

Wrapping Up

In this post, I've tried to cover the key steps from installing Pulumi and setting up your account to writing and deploying infrastructure code, all to help you get started with Pulumi.

I hope this has been helpful for those of you interested in adopting Pulumi and IaC!

References

• https://spacelift.io/blog/infrastructure-as-code-tools

• https://www.pulumi.com/docs/iac/concepts/testing/unit/

✨Enjoyed this issue?

How about this newsletter? I’d love to hear your thought about this issue in the below form!

👉 Feedback Form

Your input will be help improve Aiden’s Lab Notebook and make it more useful for you.

Thanks again for reading this notebook from Aiden’s Lab :)