$76,501.44 spent on AWS in January - A full breakdown of ConvertKit's AWS bill

generalengineeringaws
$76,501.44 spent on AWS in January - A full breakdown of ConvertKit's AWS bill
Kris Hamoud
Kris Hamoud is an Infrastructure Engineer who enjoys building simple and scalable solutions • Kris's website

Overview

We spent $76,501.44 on AWS in January. This is up 5% from December and is 4.5% of MRR in January. We spent more on EC2-Other due to data transfer costs, which also increased how much we spent on EC2-ELB.

High-level breakdown:

  1. Relational Database Service - $21,391.21 (0%)
  2. EC2-Instances - $19,352.86 (-11%)
  3. EC2-Other - $10,361.30 (+42%)
  4. S3 - $7,490.80 (+5%)
  5. Savings Plan For Compute Usage - $7,142.40 (+60%)
  6. Support - $5,123.11 (-1%)
  7. EC2-ELB - $2,356.36 (+16%)

Relational Database Service - $21,391.21 (0%)

We terminated the temporary database we created in December part of the way through January. We also stopped with the unnecessary backups of that database, which saved us about $100/day on RDS:ChargedBackupUsage costs.

Service breakdown

  1. USE2-HeavyUsage:db.r5.12xl - $4,949.68 (0%)
    • This instance is reserved.
    • This is our master MySQL database.
    • We’ll continue to pay this much until Q3 2020.
  2. RDS:ChargedBackupUsage - $4,586.07 (+4%)
    • These are our disaster recovery backups.
    • We take additional backups and send them to a different region in case of emergencies.
    • The increase comes from the extra backups taken by the temporary database we were running. We turned off the unnecessary backups, and our daily costs reduced by $60/day. RDS:ChargedBackupUsage
  3. USE2-InstanceUsage:db.r4.8xlarge - $2,856.96 (0%)
    • This is an on-demand instance.
    • This replica is being kept around because we need it to maintain a healthy application for the foreseeable future.
    • The price is equal because January is the same length as December.
  4. USE2-RDS:Multi-AZ-GP2-Storage - $2,119.22 (0%)
    • These are daily charges.
    • The cost is equal because January and December are equal lengths, and we haven’t added any storage since November.
  5. USE2-RDS:ChargedBackupUsage - $1,709.46 (-24%)
    • These are our normal backups.
    • The decrease is because we were running backups on the temporary database.
    • After getting rid of the unnecessary backups, this daily charge increased, but because we lowered our backup retention, it is still cheaper than it was in December by about $30/day. USE2-RDS:ChargedBackupUsage
  6. USE2-RDS:GP2-Storage - $2,005.96 (+11%)
    • This is the cost of our storage.
    • The increase comes from the temporary database we were running through most of January
    • When we deleted that database this daily cost dropped by about $30/day. USE2-RDS:GP2-Storage
  7. USE2-HeavyUsage:db.r4.8xlarge - $1,649.89 (0%)
    • This instance is reserved.
    • This is our MySQL replica.
    • We’ll continue to pay this much until Q3 2020.
    • The price is equal because January and December are equal lengths.

EC2-Instances - $19,352.86 (-11%)

We purchased a Savings Plan For Compute Usage in December, which put our EC2-Instances bill under $20k/month for the first time since August. Most of our EC2-Instances line items either decreased or stayed flat. The amount we spent on EC2-Instances in January is over $7,000 less than we spent at our peak in November. The only large increase came from USE2-HeavyUsage:i3en.2xlarge. These costs increased because we switched our primary storage on our ElasticStack from i3.2xlarge instances to i3en.2xlarge instances for their cheaper cost per gigabyte of storage.

Service breakdown

  1. USE2-HeavyUsage:i3.2xlarge - $8,916.10 (+1%)
    • These are our reserved Cassandra and Elasticsearch clusters.
    • We use Cassandra to store massive amounts of data.
    • We use Elasticsearch to search through massive amounts of data and to store our logs.
    • We stopped adding these instance types to our Elastic Stack in favor of the storage optimized i3en.2xlarge instances that come with 5TB total SSD storage.
  2. HeavyUsage:i3.2xlarge - $2,865.89 (0%)
    • These are reserved Cassandra instances.
    • We use them for our secondary Cassandra cluster.
    • The price is higher because December is longer than November.
  3. USE2-HeavyUsage:c5.2xlarge - $1,928.45 (0%)
    • These are reserved instances.
    • We use these for our web servers.
    • We’ll pay this much until Q3 2020.
    • The price is equal because January is the same length as December.
  4. USE2-DataTransfer-Out-Bytes - $1,625.99 (0%)
    • This is the cost of our services to communicate with the internet.
    • We had our first full month of billing wins here because we migrated away from our old logging provider in December.
    • Our daily costs are all over the place because our service usage varies throughout the week. USE2-DataTransfer-Out-Bytes
  5. USE2-BoxUsage:c5.2xlarge - $1,345.49 (-55%)
    • These are on-demand instances.
    • The decrease comes from our Savings Plan For Compute Usage
    • Despite the large decrease, we can see a steady increase in costs due to increased usage of our savings plan. USE2-BoxUsage:c5.2xlarge
  6. USE2-HeavyUsage:i3en.2xlarge - $1,245.57 (+315%)
    • These are reserved instances.
    • We switched from i3.2xlarge to the i3en.2xlarge because they are cheaper per gigabyte of storage.
    • We’ll pay this much until we need to scale out again to meet our storage needs.
  7. Others - $1,425.38 (-68%)
    • This is the cost of every other instance type that we run.
    • If we can keep our on demand usage down, this number shouldn’t change too much from month to month.

EC2-Other - $10,361.30 (+42%)

The majority of this bill is our inter-region data transfer. Because our ElasticStack instances aren’t sized equally, they have been rebalancing shards between our i3.2xlarge and i3en.2xlarge instances, which has increased our data transfer pretty significantly. We’ll be able to fix this in March when our first i3.2xlarge reservations begin to expire.

Service breakdown

  1. USE2-DataTransfer-Regional-Bytes - $6,332.56 (+94%)
    • This is the cost of replicating data in our data stores across AWS regions.
    • The cost increased here because of a logging change we shipped on December 9th. USE2-DataTransfer-Regional-Bytes
  2. USE2-NatGateway-Bytes - $1,369.70 (-18%)
    • We use a NAT gateway for our services to communicate with the internet.
    • We saw a decrease here after removing our third party logging integration. USE2-NatGateway-Bytes
  3. USE2-EBS:VolumeUsage.gp2 - $1,463.23 (+7%)
    • This is the cost of having gp2 disks connected to our instances.
    • The number of these was flat for most of January although we had a small increase for a few days at the beginning of the month. USE2-EBS:VolumeUsage.gp2

S3 - $7,490.80 (+5%)

We had a few days in early January, where our data transfer costs had increased because we finished migrating objects from one storage solution to another. During that time, it took about two days for our CDN to fill up before we got back to downloading primarily from cache.

Service breakdown

  1. DataTransfer-Out-Bytes - $2,047.04 (+1%)
    • We continue to see billing wins here as the number of objects being served straight from S3 decreases in favor of our Cloudflare CDN.
    • The slope continues downward despite the two days in early January where it increased. DataTransfer-Out-Bytes
  2. USE2-TimedStorage-ByteHrs - $1,852.86 (+8%)
    • This is the steady growth of our backups coming from our Cassandra and Elasticsearch clusters.
    • This will continue to grow as the amount of data we store grows.
    • We can probably find wins here by changing our storage class to infrequent access.
  3. USE2-DataTransfer-Out-Bytes - $1,654.46 (+11%)
    • We served more data from our Cloudflare CDN here than I was anticipating.
    • Aside from a couple of days in the middle of the month, our daily data transfer was equal to most of November.
  4. TimedStorage-ByteHrs - $1,106.74 (-2%)
    • This decrease comes from a reduction in retention length of our backups.
    • It will continue to grow as the amount of data we store grows.

Support - $5,123.11 (-1%)

Service breakdown

  1. 7% of monthly AWS usage from $10K-$80K - $3,123.11 (-7%)
    • This is the cost of only our production account.
  2. 10% of monthly AWS usage for the first $0-$10K - $2,000.00 (+11%)
    • This is the cost of our production account and billing account.
    • We could save money by turning off support for our billing account.

Savings Plan For Compute Usage - $7,142.40 (+60%)

We saved $2,704.00 in January from purchasing this. Savings Plan For Compute Usage

EC2-ELB - $2,356.36 (+16%)

We use load balancers for many different parts of our application. The cost has been increasing steadily over the past few months as we add more autoscaling groups to our infrastructure, handle more requests to our Free Landing Pages, and increase our log volume. We can find wins here in the future as we migrate to Kubernetes and can rely on the internal networking of Kubernetes rather than AWS resources.

Service breakdown

  1. USE2-DataTransfer-Out-Bytes - $1,061.01 (+8%)
    • This is the cost of data going out to the internet from behind our load balancers.
    • This increase comes from higher use of our application in January than in December.
  2. USE2-LoadBalancerUsage - $630.87 (+13%)
    • This cost comes from the number of connections to our load balancers and their overall usage.
    • This cost is steadily increasing as we handle more traffic to our landing pages.
  3. USE2-LCUUsage - $626.85 (+16%)
    • This increase comes with the EC2 data transfer increase we saw beginning in December.
    • We’ll see wins here in the future as we migrate more of our workload to Kubernetes and can use internal load balancers instead of AWS provisioned load balancers.

Conclusion

We’re continuing to save money from our Savings Plan For Compute Usage, so our EC2-Instances costs shouldn’t grow too quickly. We got rid of our extra database and backups, so our Relational Database Service costs should decrease in the coming months. We have a data transfer problem that we need to solve, causing our EC2-Other to increase quickly. Our EC2-ELB costs are increasing too, but this should be solved as more of our services migrate to Kubernetes. Despite our January bill being more expensive than December, we’re getting better at predicting future costs and predicting future problems so we can work towards solutions.