Skip to main content
Ctrl+K

docs

  • Twitter
  • Mastodon
  • Contact
  • Blog
  • Twitter
  • Mastodon
  • Contact
  • Blog

Infrastructure Guide

Get started

  • Documentation structure
  • Setting up your local environment to work on this repo

SRE guide

  • Support tasks
    • Access home directories of users of a hub
    • Add a simple python package to an image we maintain
    • Decrypt encrypted information sent to support@2i2c.org
    • Apply Cloud Credits
    • Give Grafana access to a community representative
    • User access to Prometheus endpoint
    • Adding admin users based on a support request
    • Provide the hub on a community maintained domain
    • Reset password for shared password hubs
    • Reset a Hub
  • Manage cloud and Kubernetes infrastructure
    • Authenticating with cloud providers for 2i2c engineers
    • Manual node administration
    • Cull resources
  • Scaling nodepools
    • AWS
    • Azure
  • Common problems and their solutions
  • Resizing Prometheus’ disk
  • How to upgrade the versions of the support chart’s dependencies

Hub deployment guide

  • Runbooks on how to handle new hub turn-up requests
    • Phase 1: Account setup
      • Create a new GCP project
      • Create a new AWS account
      • AWS with external account
    • Phase 2: Cluster setup
    • Phase 3: Hub setup
      • Phase 3.1: Initial setup
  • Add Kubernetes clusters
    • New Kubernetes cluster on GCP, Azure or AWS
    • AWS with NASA SMCE
  • Deploy support components
    • Configure and deploy the support chart
    • Setup grafana dashboards
    • Register the cluster’s Prometheus server with the central Grafana
  • Enable authentication
    • GitHub Orgs and Teams
    • CILogon
    • Auth0
    • Canvas
    • Shared Password Authentication
  • Other hub-related operations
    • Manually deploy a config change
    • Moving Hubs
      • Move a Hub to a new URL
      • Move a Hub across clusters
    • Delete a hub

How-to guides

  • Enable specific hub features
    • Allow users to setup custom, free-form user profile choices
    • Restrict profile options based on JupyterHub groups (or GitHub teams)
    • Anonymize usernames with CILogon
    • Make binderhub-ui hub
    • Setup object storage buckets
    • Enable user access to cloud features
    • Enable stronger anti-crypto abuse features for a hub
    • Add support for daskhubs
    • Setup a dedicated nodepool for a hub on a shared cluster
    • Make an ephemeral hub
    • Allow users to push to GitHub
    • Pre-pull a content repository
    • Enable access to GPUs
    • Using JupyterHub as an identity provider
    • Enable user image features
    • Make an imagebuilding hub
    • Configure the hub login page
    • Setup a database server per user
    • Allow nbgitpuller to pull from private GitHub repos
    • Use Rocker based R images
    • Setup a shared database for all users on the hub
    • Configure per-user storage quotas
    • Integrate community-documentation with Jupyter Book
  • Upgrade Kubernetes version of clusters
    • About upgrade disruptions
    • About strategies to upgrade nodes
    • About Kubernetes’ version skew policy
    • Upgrade Kubernetes cluster on AWS
    • Upgrade Kubernetes cluster on Azure
    • Upgrade Kubernetes cluster on GKE
  • Upgrade the versions of sub-charts used in our Helm chart
  • Manage Budgets and Billing
    • Calculate how much cloud infrastructure costs
    • Setting up Budget Alerts
    • Enable cost monitoring system
      • Enable JupyterHub Cost Monitoring
  • Manage Images
    • Modify our custom JupyterHub image
    • Manage a hub’s user environment
  • Manage Filesystems
    • Migrate data across NFS servers within the same cluster
    • Transfer data between NFS servers on separate clusters
    • Filesystem Backups
      • Enable Automatic Filesystem Backups
      • Restore a Filesystem from a Backup
    • Resize a GCP Filestore down
    • Increase the size of a disk storing home directories
  • Manage events on 2i2c hubs
    • Event infrastructure preparation checklist
    • Set up a hub for an exam
  • Manage domains of hubs
    • Override a hub’s domain name
  • Manage alerts
  • Enable GitHub Organisation authentication for Grafana
  • Regenerate credentials for NASA SMDE accounts
  • Troubleshoot
    • Look at logs to troubleshoot issues
      • Cloud specific logging
      • Kubectl logging
      • Common errors and what logs to check
    • SSH into nodes
    • Troubleshoot prometheus issues
    • CILogon: switch Identity Providers or user accounts
  • Sending Secrets to communities
  • Migrate from one ingress controller to another
  • Create an GitHub workflow that’s triggered by a Pagerduty alert

Topic guides

  • Deploy a new hub
  • Access and credentials
    • Authenticating with cloud providers for 2i2c engineers
    • Secrets and private keys
  • Infrastructure
    • Configuration structure
    • Cluster design considerations
    • Terraform
    • Hub helm charts
    • Network policy
    • User home directory storage
    • Cryptnono for preventing cryptomining abuse
  • Cloud Billing
    • What exactly do cloud providers charge us for?
    • Billing accounts
    • Billing reports
    • Cloud billing tools
    • Cloud Billing Budget Alerts
  • Monitoring and alerting
    • Grafana dashboards
    • Prometheus
    • Cost Monitoring System
    • Cost Monitoring Availability
    • Alerting
    • Escalation policies and on-call
    • Simple HTTPS uptime checks
  • Features available on the hubs
  • Resource Allocation on Profile Lists
  • Jsonnet in our Infrastructure
  • Internal Tools cluster
  • Cluster Entrypoint (for HTTP & HTTPS Traffic)

Reference

  • List of running hubs
  • Our CI/CD system
    • Automatic hub deployment
    • Automatically bumping image tags and helm sub-chart versions
  • Tools used in this repo
  • CLI Reference

Contributing

  • Review and merge guidelines for 2i2c Engineers
  • Self-merging as a community partner
  • Repository
  • Open issue
  • .md

Deploy support components

Deploy support components#

2i2c have collated a selection of common tools that are deployed once per cluster and assist with actions such as load-balancing and monitoring. The following sections describe how to deploy and configure this support infrastructure.

  • Configure and deploy the support chart
    • Make sure support.values.yaml is correctly configured
    • Edit your cluster.yaml file
    • Deploy the support chart via the deployer
    • Setting DNS records
  • Setup grafana dashboards
    • Login to the cluster-specific grafana
    • Create an API key to auto-deploy the dashboards
    • Deploying the Grafana Dashboards locally
    • Deploying the Grafana Dashboards from CI/CD
    • Granting grafana access to the community representative
  • Register the cluster’s Prometheus server with the central Grafana
    • Create a support.secret.values.yaml file
    • Update your cluster.yaml file
    • Link the cluster’s Prometheus server to the central Grafana

previous

AWS with NASA SMCE

next

Configure and deploy the support chart

2i2c links

  • Twitter
  • Mastodon
  • Contact
  • Blog