Calculate how much cloud infrastructure costs

We have several community contracts where we first pay their cloud bills and then invoice them for it. Any community paying their cloud bills through 2i2c must have their hub(s) under a cluster.yaml file declaring [gcp|aws].billing.paid_by_us=true.

Practically to invoice these communities, we update a spreadsheet for billing summarizing monthly costs for communities, and then request invoicing help from CS&S via the #billing slack channel.

This documentation helps you update that spreadsheet for communities on shared clusters and generate CSV files for dedicated clusters.

Communities in shared cloud accounts¶

We used to run hubs on a shared cluster, but we don’t anymore. There’s only one hub remaining on shared clusters - Michigan Tech University. We will migrate it to its own cluster at some point soon. Until then, since it’s the only hub running on the two-eye-two-see pilot-hubs cluster, we can use this report to roughly approximate the cloud cost of the MTU hub.

Communities with dedicated cloud accounts¶

Get a community dedicated AWS accounts’s costs¶

1. Login to shared AWS SSO at https://2i2c.awsapps.com/start#/

2. Select the 2i2c-sandbox account, as it is the primary billing account

3. Select ‘AdministratorAccess’ to open the AWS Console for this account

4. Visit the “Monthly Costs By Linked Account” report (direct link) via “Billing and Cost Management” -> “Cost Explorer Saved Reports”

5. On the right sidebar under “Time -> Date Range”, select all the completed months we want to get data for

6. On the right sidebar under “Time -> Granularity”, ensure its selected as “Monthly”

7. On the right sidebar under “Group by -> Dimension”, select “Linked account”

8. Click the ‘Download as CSV’ button

   

9. Run the following deployer command to convert the generated CSV file into the format required for the invoicing process.

   deployer transform cost-table aws pathto/downloaded/csvfile

   This will output a new CSV file to your local filesystem called AWS_{START_MONTH}_{END_MONTH}.csv.

10. Upload this CSV file to the cloud costs folder

11. As of July 2024, ping James and Harold in the #billing slack channel to let them know the info for dedicated clusters is now available and provide a link to the file you have just uploaded

Get a community dedicated GCP projects’ costs¶

1. Go to the 2i2c billing account on GCP

2. Select ‘Reports’ on the left sidebar

3. Under time range on the right sidebar, select ‘Invoice Month’

4. Select the time range you are interested in. Note that this has to be at least two months right now, or the next step does not work

5. Under ‘Group by’, select ‘Month -> Project’ or just ‘Project’ if the time range is only one month

6. Under the chart, click the ‘Download CSV’ button.

   

7. Run the following deployer command to convert the generated CSV file into the format required for the invoicing process.

   deployer transform cost-table gcp pathto/downloaded/csvfile

   This will output a new CSV file to your local filesystem called GCP_{START_MONTH}_{END_MONTH}.csv.

8. Upload this CSV file to the cloud costs folder

9. As of July 2024, ping James and Harold in the #billing slack channel to let them know the info for dedicated clusters is now available and provide a link to the file just uploaded

Get a community’s costs from OVH¶

We currently have an account on OVH Cloud US, where we pay for a mybinder.org federation member for BIDS.

1. Go to [https://manager.us.ovhcloud.com/#/billing/history](OVH US Billing Console) using credentials provided to you. If you don’t have credentials, reach out to Yuvi to get some.

2. Under filter, select:

   1. “Issue Date” as column

   2. “is after” as condition

   3. The start of the month for which we are looking at cloud costs as “Value”. For example, if you’re looking for cloud costs for January 2026, set this value to 1 Jan 2026. The goal is to make sure we get all the invoices that have been issued since our last time, so verify by looking at the last OVH invoices upload.

3. Under “Actions” click “Export as CSV”.

4. Go to the Cloud Costs Google Drive Folder and upload this CSV. Rename it to OVH_<starting-year>-<starting-month>-<starting-date>_<ending-year>-<ending-month>-<ending-date>, based on the first and last entry in the invoice (eg. OVH_2025-12-01_2026-02-01.csv)

5. Post a link to this in the #billing channel

Experimental¶

We have an unfinished attempt to automate collection of community monthly costs. This heading retains documentation about that, but we are for 2024 not driving development here and instead relying on the manual approaches.

Get costs automatically via the deployer (not ready to be used yet!)¶

The generate cost-table subcommand of the deployer will go through all our clusters set up on Google Cloud, and tell you how much they cost.

Pre-requisites for running it:

1. You have all tools required required to work on this repo setup.

2. Run gcloud application-default auth login, and authenticate with your 2i2c.org google account. This account must have permissions to all GCP projects. This requirement will be relaxed at some point in the future.

There is a private Google Sheet that has monthly costs for all the clusters that are configured to have bigquery export.

This sheet is currently manually updated. You can update it by running deployer generate cost-table --output 'google-sheet'. It will by default update the sheet to provide information for the last 12 months. You can control the period by passing in the start_month and end_month parameters.

If you just want to take a look at the costs in the terminal, you can also run deployer generate cost-table --output 'terminal' instead.

Caveats¶

1. The data comes from bigquery costs export, so is only available and accurate after that has been enabled. For billing data before this was enabled, you need to manually go look in the cloud console.

Other Cloud Cost Reporting Exporting¶

• AWS Cost and Usage Report

• Azure Cost Export