“And if we can’t experiment, we can’t learn!”
Gene Kim, The Unicorn Project
Everybody has to learn at least once. Like putting your hand onto a hot stove as a kid (despite being told it’s a bad idea), to popping a balloon in your face, or getting a small cylinder stuck into a mini M&M’s tube; learning is part of life.
I started my infrastructure journey late last year, working for a small company down under known as Trade Me. I interned with their payment processing team as a cloud infrastructure engineer.
Before this, I had almost no experience with the cloud, apart from a few projects done in high school and university, mainly with Azure and GCP. This was like jumping into the deep end, on top of moving out of home to a different city, on the other side of the country; I didn’t even know if I would enjoy the work.
Fortunately, I did enjoy the work. I REALLY enjoyed the work.
I’ve talked with a few people about what made them love working with the cloud. The main consensus that I was able to take away is that every day is a new challenge and new things to learn, which I agree with 100%.
The Power of the Cloud
“With the cloud, individuals… can snap their fingers and instantly set up enterprise-class services”
Roy Stephan – PierceMatrix
This quote highlights how amazing the cloud can be. It is amazing to be able to spin up and down servers in a matter of minutes and have complete control over their network configuration. But like all nice things, there is a cost—some more than others.
Today, creating the same services and resources with providers such as AWS has become easier than it was, say, 20 years ago. Management of virtualised servers and systems was left to the experienced; no new person could touch it with a 10-foot pole. Today, though, we have resources and frameworks such as Terraform and AWS CDK, which have made it 1000 times easier to create, manage, and destroy infrastructure.
Today, it can be as easy as writing a few lines of code to spin up a new VM. In Terraform, it’s as simple as:
resource "google_compute_instance" "vm_instance" {
name = "wordpress-server"
machine_type = "e2-micro" # <-- Machine = Money
network_interface {
network = "default"
access_config {
network_tier = "STANDARD"
}
}
boot_disk {
auto_delete = false
initialize_params {
image = "ubuntu-minimal-2204-jammy-v20250502"
size = 30
type = "pd-standard"
}
}
tags = ["web", "http-server", "https-server"]
metadata_startup_script = <<EOF
...
EOF
}But this also means it can be just as easy to misconfigure something. Something as simple as one line can cost an organisation a little bit more than they usually expect, but for an individual, it could be an increase of 1000% from last month’s billing.
For example this GCP machine type cost $84.22 USD/H
machine_type = "m2-ultramem-416"Whereas the previous example is covered under Google’s free tier, thus practically $0 USD/H.
My Introduction to AWS CDK
My first introduction to CDK was a project I was given from a job’s final round of interviews. I was given a week to “create a CMS application using AWS’s Cloud Development Kit (CDK).” I had never used CDK before this, only Terraform. Further, working full-time did not help, as I had no time to study.
Fortunately, the AWS CDK has a library for Python, which I have learned and tutored in the past. Not to say that I’m a guru at Python (which I’m not), but I found it easy to pick up the CDK, as most of the things I learned using Terraform were also applicable, albeit with a different way of defining the infrastructure. And this is the same with all programming, you learn it once, be it C or Python, and it becomes way easier to learn other languages such as Java or Go (but I still can’t figure out Rust).
For me, as I mentioned before, I love to learn new things, so I said screw it. Since I was given an AWS account to use for the project, I wasn’t the one to foot the bill, so it couldn’t hurt to try new things.
The infrastructure was as below:
It’s a pretty simple and straightforward infrastructure, using containers and implementing serverless services that can automatically increase or decrease their resources on the fly.
And the best thing, it worked. Until I made the mistake that everyone experiences at least once.
Enterprise vs. Individuals
I think this section is simple enough to breeze over. Enterprise environments are built in such a way for a reason. They are designed with security, availability, and scalability in mind. The CIA principles and ISO standards that the security team lives by, always looming, influencing every change and decision in an organization.
Individuals on the other hand do not have such worries, and so infrastructure should not be built with the same concepts in mind. Sure security, availability and scalability should not be entirely brushed aside, but if I were to choose between high availability and $30, I would choose the $30.
“Two Dollars Per Change Please”
This is where things get interesting. As I discussed misconfigurations are the bane of everyone’s existence.
After completing the project, I decided that I could re-use the code that I wrote. I was already looking to properly host my own website rather than piggy backing of GitHub pages. My website was in a dire need of a redesign; which the result being this WordPress site that you are on now! (This time using GCP rather than AWS, and not being containerized).
I had already had done a bit of work beforehand using the Serverless framework to host a Laravel site using AWS Lambda functions, but I thought that since I know that the code that I have now already worked, why not just use it.
This was my mistake…
Such a design would be fine in an “enterprise” environment, but it was completely overkill for a “individual” project.
I thought that I had designed the infrastructure in such a way to be “cost-effective”, but once again it just takes one misconfiguration or mistake to blow everything up.
My mistake?
Destroying and rebuilding my CDK CloudFormation RDS Stack multiple times during development.
I needed to destroy the stack during my development phase so I could reset the WordPress configuration. In hindsight, I did not need to do this as I could have just deleted the WordPress related tables, but that was no fun (at the time).
In short, one day of development costed me $30 USD.
The kicker, the next month’s cost estimate was at $500 USD.
I quickly shut down both the RDS and ECS Stacks after reading that number.
“Leave and Forget”
Like the original “Spray and Walk Away” adverts of the mid 2000s, like with cloud resources you can sometimes forget you have things running.
After the $500 potential bill scare, as any good person should do, I checked again on the Cost and Billing Management center, using my root account. I did this a few days later to see what my new estimate was.
$140 USD.
I couldn’t believe my eyes. I thought that I had shut down the main perpetrators of the massive costs. Unfortunately not.
This goes back to the point of Enterprise vs. Individual. At every time you decide to create some infrastructure you need to consider the necessity of such a service or resource.
In my case my VPC structure was the perpetrator.
According to AWS documentation, each VPC endpoint per AZ per hour costs $0.013. I had multiple VPC endpoints per Availability Zone, thus I was passively being charged every hour for endpoints that had no traffic going through them (since I panic destroyed every stack – apart from my VPC stack).
Security vs. Financial Security
In hindsight, I messed up from the very beginning. I screwed up in the planning stages of my infrastructure. I failed to properly recognize who the real client was; myself.
Did I really need high availability with services like Aurora? No.
Did I really need the security which comes with VPCs, private domains and multiple AZs? Definitely not.
Normally I would say security is extremely important since coming from a security background, but now I’m definitely more lenient when it jeopardizes my own financial security.
Conclusion
After a series of bad mistakes and lessons to be learned, like my AWS friend said “to only use AWS when you really need to”, I’m going to listen to his advice.
Which is why I am now using GCP to host this site…
The main things that I learned from my missadventures:
- Always check the billing dashboard before it surprises you.
- Cloud resources are deceptively easy to deploy—and just as easy to forget.
- Enterprise-level designs are tempting, but unnecessary for personal projects.