< cd ../blog
Jun 15#aws#infrastructure

AWS Things Worth Knowing

The AWS manual is written for a generic org. The lessons that actually matter live in scar tissue — the ones people only share after they've paid for them. What follows is a curated read of an r/aws thread where practitioners traded those lessons. The non-obvious ones. The ones that change architectures.

Data transfer is the silent killer

Every network boundary crossing costs money. Ingress is free; egress will wreck your bill. One commenter's network spend was 30% of their total AWS cost.

A few specifics worth designing around:

  • VPCs are non-transitive. Basic IP routing has done transit since the 1970s. AWS welded the limitation in and sells you Transit Gateway, billed by the byte.
  • NLBs charge for cross-AZ transfer. ALBs don't. That single distinction can change your architecture for high-throughput workloads.
  • NAT Gateways eat giant holes in your wallet. Alternatives exist, all with their own drawbacks.
  • VPC endpoints and PrivateLink keep traffic off the internet path and out of the metered crossings.

If you're fully migrating off AWS or shutting down accounts, AWS will waive egress fees. They'll let go — but only on the way out.

The bill is the architecture diagram

A few items that quietly compound:

  • Stopped EC2 still bills you for the attached EBS volumes. "Stopped" isn't free.
  • T-instances aren't always a deal. Burstable sounds great until hypervisor over-provisioning makes your latency unpredictable. For sustained workloads, M or C instances often win on actual cost.
  • Spot is up to 90% cheaper than on-demand and rarely capacity-starved. It should be the default for non-prod.
  • Tag everything. Then add an Expiration tag in RFC3339 and let a janitor Lambda flag or destroy what's past its date.
  • Pre-sales engineer estimates run 25–50% lighter than reality. They're rushed and they miss things. Plan for the larger number.

Send daily cost reports to team leads. Public accountability curbs runaway services faster than any FinOps deck.

Temporary is permanent

The dev who says "this VM is just a test, it won't last long" is lying — sometimes to themselves. That VM will live until the end of time.

Document it like prod from minute one. Back it up. Be ready to rebuild it on the worst possible day, because that's the day it will die.

IAM and the unkind defaults

IAM rewards careful reading and punishes assumption.

  • Don't end Secrets Manager secret names with -XXXXXX (dash plus exactly six characters). AWS treats the trailing six as the random ARN suffix it generates. Cross-account access silently breaks. Five characters or seven is fine.
  • Lock the root account away. Unique password in a vault, two physical MFAs in two physical locations, optional software MFAs. Then never use it.
  • Have break-glass admin users for a few key people, MFA-protected, with alerting on use. Rotate their passwords periodically as a fire drill.
  • KMS policies draw boundaries around your data; IAM policies govern your actions. Trying to protect data by blocking API calls on parameters is a losing battle.
  • Start an EC2 instance whose EBS volumes are encrypted with a KMS key you can't decrypt and the API call returns success — but the instance never actually starts. Confusing the first time. Embarrassing the second.

Accidental databases

The most entertaining part of the thread is the catalog of services people have abused into being databases.

  • Route 53 TXT records as a key-value store. Not a joke. People ship this. There's a whole post about it.
  • EventBridge schedule descriptions storing dictionaries. One commenter moved a project off DynamoDB into EventBridge schedule descriptions and watched the bill drop from $100/month to nearly zero.
  • DynamoDB is a great choice only if you're 100% certain you'll never need many-to-many relationships or analytics. RDS isn't the world's most expensive database — it's the world's cheapest DBA.
  • S3 is a key-value store, not a filesystem (Mountpoint notwithstanding). Treat it like one and you'll stop fighting it.

Operational footguns

The kind of thing that costs an afternoon the first time and ten minutes every time after:

  • S3 failed multipart uploads are invisible in the bucket, never auto-delete, and you pay for the storage. Set a lifecycle rule once, forget about it forever.
  • Mac instances have a 24-hour minimum commitment. That mistake costs about $50 each time someone learns.
  • ACM will accept a cert chain in the wrong order without complaint. Browsers silently fix it. CloudFront does not. Your ALB will look fine while CloudFront serves errors. Verify with whatsmychaincert.com.
  • CloudWatch Logs regex needs %...% delimiters. Not slashes. Not quotes.
  • EBS volume resize is limited to 4 per rolling 24 hours. Don't be stingy with the initial size.
  • Transit Gateway cross-account attachments must use the same AZ ID (euw2-az2), not the AZ name (eu-west-2a). Names are randomized per account; IDs are not.

Well-Architected is a guide, not a rulebook

AWS wrote it for a generic org. Your constraints aren't generic. Cherry-pick what applies. Sometimes the framework's recommendation translates to "well-architected billing."

The meta-lesson

AWS rewards two habits and punishes their absence: reading the boto3 docs for the API call that creates a thing, and expressing infrastructure as code. Clickops and assumption are the expensive paths. Almost every story in the thread reduces to a moment when one of those two habits would have saved someone a bill, an outage, or an afternoon.

The lessons aren't in the manual. They're in the postmortems.