Calculate how much cloud infrastructure costs#
We have several community contracts where we first pay their cloud bills and
then invoice them for it. Any community paying their cloud bills through 2i2c
must have their hub(s) under a cluster.yaml file declaring
[gcp|aws].billing.paid_by_us=true.
Practically to invoice these communities, we update a spreadsheet for billing
summarizing monthly costs for communities, and then request invoicing help from
CS&S via the #billing slack channel.
This documentation helps you update that spreadsheet!
Important
Make sure you have access to the spreadsheet for billing before following the steps below. Your 2i2c.org email should give you access.
AWS#
Get costs manually via the UI (only documented option)#
Note
As of 2024-04-17, we only manage the cost for AWS accounts associated with our
AWS management account. If a future cluster deviates from this, you can tell by
aws.billing.paid_by_us being set true in its cluster.yaml.
Login to shared AWS SSO at https://2i2c.awsapps.com/start#/
Select the
2i2c-sandboxaccount, as it is the primary billing accountSelect ‘AdministratorAccess’ to open the AWS Console for this account
Visit the “Monthly Costs By Linked Account” report (direct link) via “Billing and Cost Management” -> “Cost Explorer Saved Reports”
On the right sidebar under “Time -> Date Range”, select all the completed months we want to get data for
On the right sidebar under “Time -> Granularity”, ensure its selected as “Monthly”
Click the ‘Download as CSV’ button
Copy AWS costs
The CSV file has rows for each month, and columns for each project. Copy it into the spreadsheet, making sure the rows and columns both match what is already present. The AWS account name should match the “Cloud Project” row in the spreadsheet.
Once done, re-check the numbers once again to make sure they were copied correctly.
GCP#
Get costs manually via the UI (recommended)#
Important
Currently this is the recommended way of retrieving the costs from GCP.
Go to the 2i2c billing account on GCP
Select ‘Reports’ on the left sidebar
Under time range on the right sidebar, select ‘Invoice Month’
Select the time range you are interested in. Note that this has to be at least two months right now, or the next step does not work
Under ‘Group by’, select ‘Month -> Project’.
Under the chart, click the ‘Download CSV’ button. This downloads a CSV that you can use to later populate the columns in the costs spreadsheet
GCP billing UI#
Copy GCP costs.
The GCP CSV has rows for each month and project. Carefully copy the $ values under ‘subtotal’ (last column) into the spreadsheet for billing. Match the ‘project name’ column in the GCP CSV with the ‘cloud project’ row in our billing spreadsheet.
When done, re-check the numbers once again to make sure they are copied correctly.
Get costs automatically via the deployer (not ready to be used yet!)#
Warning
This deployer command requires more development work and is not yet recommended to be used.
The generate cost-table subcommand of the deployer will go through all our
clusters set up on Google Cloud, and tell you how much they cost.
Pre-requisites for running it:
You have all tools required required to work on this repo setup.
Run
gcloud application-default auth login, and authenticate with your2i2c.orggoogle account. This account must have permissions to all GCP projects. This requirement will be relaxed at some point in the future.
There is a private Google Sheet that has monthly costs for all the clusters that are configured to have bigquery export.
This sheet is currently manually updated. You can update it by running
deployer generate cost-table --output 'google-sheet'. It will by default
update the sheet to provide information for the last 12 months. You can control
the period by passing in the start_month and end_month parameters.
If you just want to take a look at the costs in the terminal, you can also run
deployer generate cost-table --output 'terminal' instead.
Warning
If the script is run before the end of the month, the total costs would not be accurate. Run the script once the month has finished to get an accurate amount of the previous month’s costs.
Caveats#
The data comes from bigquery costs export, so is only available and accurate after that has been enabled. For billing data before this was enabled, you need to manually go look in the cloud console.