Georgiana Dolocan | 2i2c

Upgrading community infrastructure to Kubernetes 1.34 and JupyterHub 4.3.3

Wed, 08 Apr 2026 00:00:00 +0000

We’ve completed a major round of infrastructure upgrades across all 2i2c-managed hubs - every hub is now running Kubernetes 1.34 and Z2JH helm chart 4.3.3.

Running up-to-date versions of both Kubernetes and the JupyterHub helm chart ensures that our communities get the best support and reliability, both in terms of features and security.

A new approach to infrastructure upgrades: upgrading in rounds #

This was the first time we rolled out JupyterHub helm chart upgrades in rounds rather than all at once. By upgrading a subset of hubs at a time, we could identify and fix issues in isolation before they affected the broader network. This made the process safer and more predictable.

We’re planning to perform these kinds of upgrades on a regular schedule for our member communities. Around every 6 months we’ll create an issue to make sure nothing falls through the cracks (here’s example config for creating our reminder issues).

Check out our process docs for multi-hub upgrades for more information.

Learn more #

Check out these pages for what kinds of improvements we’ve brought into our clusters / hubs with these latest updates.

Acknowledgements #

Thanks to Georgiana Dolocan for leading this upgrade effort and establishing the rounds-based approach.
Thanks to Chris Holdgraf for adapting and editing Georgiana’s notes into a blog post.

Improving our community hub reliability and stability in Q4 2025

Tue, 16 Dec 2025 00:00:00 +0000

This year we’ve prioritized making the cloud safe to try for our member communities. This has driven work in monitoring, alerting, and automating infrastructure so that we resolve small problems before they become big problems. In the last quarter of 2025, we wrapped up this effort by testing the following hypothesis:

We can reduce P1 incidents if we shorten the time to act on current alerts and learnings from prior incidents.

Here’s what we accomplished and what we learned.

What we accomplished #

In short: we’re now much more confident in the stability of community infrastructure. Here’s a snapshot of our new incident dashboard, which shows high-level trends for the stability of our infrastructure:

See the real-time status of our community hubs at status.2i2c.org

We improved infrastructure reliability for our communities #

We made several technology and team process improvements that led to these benefits for our communities:

We are now more likely to catch outages before a community reports them to us.
We are now less likely to have an outage happen more than once, or affect more than one community, because we consistently fix the issues that cause outages.

We saw a consistent drop in critical alerts that required immediate response:

For August and September we had an average of 7 outages/month (6 from alerts, 1 from community)
In October, November, and December we had an average of 3 outages/month (9 in October, 0 in November, 1 in December, with only one of these being reported by a community)

We became more efficient, responsive, and focused #

We also got several team benefits from this work:

We get fewer interruptions and distractions from deeper work.
We have clear assignment policies to make it clear who is responsible for acting in response to alerts.
We avoid invisible work from falling down rabbit-holes when responding to outages.
We decreased the stress and pressure of doing upgrades, making them easier to split into sprint items and more likely to get done consistently.

The improvements we made #

Infrastructure improvements #

Created a status page for all 2i2c community hubs, giving our team and communities visibility into the status of our infrastructure.
Created an alert that triggers when two servers fail to start consecutively in a 30-minute time window.
Improved deployment infrastructure so that we can roll out sub-chart upgrades to individual clusters, allowing us to roll out major changes in batches.
Removed our “configurator” application from community hubs, because it was causing more confusion than it was resolving.
Allowed servers to start even when users hit their storage quotas.
Provided a number of upgrades to Kubernetes and the support services that we run alongside each community hub.

Process improvements #

Made a team commitment to prioritize issues from incident reports and other stability-related problems.
Defined incident escalation policies using the status page to calibrate the urgency of our response to the severity of incidents.
Defined “on-call” procedures so our team knows when and how to be more responsive to outages.
Time-boxed our alert response process to avoid accidentally falling down rabbit holes for non-urgent problems.
Created a more reliable process for responding to incidents and writing incident reports.

Looking forward #

After this push around infrastructure reliability, we’re significantly more confident in the stability and transparency of our community hub infrastructure. This will deliver better service for our member communities and free up more of our time to engage with them instead of fighting infrastructure fires.

We will continue to improve our infrastructure, and have a better foundation to do so incrementally in the coming quarters. Here are a few things we’d still like to improve:

We still need to improve how reliably we complete follow-up actions from incidents (e.g., writing incident reports). When a process doesn’t fit into planning & scoping ceremonies, we struggle to follow it consistently.
We’d like to improve our testing framework for major upgrades across all hubs (e.g., Kubernetes version upgrades) to catch bugs before communities do.

Learn More #

Offering Jetstream2-powered hub support at 2i2c

Mon, 28 Apr 2025 00:00:00 +0000

When we first committed to offer Jetstream2 support at 2i2c, Jetstream2, Magnum, OpenStack, ClusterAPI were all new concepts that we hadn’t used at 2i2c before. And although the initial exercise of reading about each of them independently was confusing, learning how they actually glued together was the key. This post is about Jetstream2, 2i2c persistent hub offerings, and the learning that took place in the process.

⭐ Members of 2i2c’s community network can determine their eligibility and learn about JetStream2 in our supported cloud providers documentation. If needed, reach out to 2i2c for support.

Context #

At 2i2c, we want to be able to deploy k8s clusters on different cloud providers. In a very simplistic way, for this we use:

Infrastructure as code to describe, deploy and manage the actual physical infrastructure from the cloud providers
Cloud specific CLI to authenticate to this infrastructure
Helm to deploy and manage k8s resources onto this infrastructure
And finally kubectl to interact with all of these k8s resources

(Main tools used at 2i2c to deploy and manage k8s clusters on different cloud providers)

On cloud providers like GCP, AWS, Azure, the Kubernetes support feels like an atomic feature of the cloud provider and works out of the box. But on Jetstream2, k8s support is not such a solid feature anymore.

Jetstream2 Kubernetes support stack #

Jetstream2 is a collection of supercomputers that are part of the ACCESS cyberinfrastructure. This ACCESS infrastructure groups together super computers like Jetstream2 (but not limited to it), into a mesh that creates the impression of a single, virtual system that scientists can openly access and interactively use.

It offers Infrastructure as a Service (IaaS), that allows users to deploy VMs and manage environments dynamically. And the piece that enables this Infrastructure as a Service feature is OpenStack.

OpenStack and Magnum #

OpenStack is an open source platform made of multiple projects that help build and manage both private and public cloud infrastructure.

For our use-case, one of the most relevant OpenStack sub-project is Magnum. Magnum offers container orchestration engines for deploying and managing containers, like Kubernetes, but not limited to it.

Initially, Kubernetes support was provided through a project called HEAT. However that has proven harder to manage and maintain, and it was extremely hard to upgrade a cluster. So, they’ve migrated towards a new driver called Cluster API magnum driver, which offers a more native k8s integration.

Cluster API and CAPI helm driver #

CAPI itself is k8s project that allows declaring k8s clusters in an easy way.

The helm driver on the other hand is what acts like a bridge between OpenStack’s Magnum and Kubernetes’ Cluster API (CAPI). Its main goal is to to manage the lifecycle (create, scale, upgrade, destroy) of Kubernetes-conformant clusters using a declarative API.

In order to do this, Cluster API provides an API for being able to manage the various components of a Kubernetes cluster. This conceptually looks like a Kubernetes cluster managing other Kubernetes clusters; the former, named the ‘CAPI management cluster’, is the one providing the API for managing the latter workload clusters.

Decomposing the previous atomic feature #

(Comparison between Jetstream2 and other cloud providers when it comes to k8s support)

Magnum is part of the OpenStack tent and it’s the first layer on top of Jetstream2 towards achieving k8s support.

The CAPI helm driver is what’s offering CAPI support. This is the last piece that’s needed to link a k8s cluster down to the hardware where it’s deployed, on Jetstream2.

Challenges #

The Jetstream2-OpenStack stack is not a simple one. It’s a complex stack of technologies and each of the connection points can be challenging to debug and fix when something doesn’t work. Especially when you are one of the first ones that pilots this new magnum driver setup.

So, it was expected that we faced some issues along the way. However, we were able to go around them and add Jetstream2 to our service menu. Below is a list of some of the issues that we faced:

We have to create terraform resource in sequence which takes longer because of a race condition that makes concurrent nodegroups creation requests to fail

bugs.launchpad.net/magnum/+bug/2097946

The role and labels of the nodegroups don’t get propagated to the actual nodes, so we cannot put our own labels on nodes at once

azimuth-cloud/capi-helm-charts#84

The node count and min node count cannot be set to 0 and each nodegroup has to have at least 1 node

bugs.launchpad.net/magnum/+bug/2098002

A default-worker is created apart from the default-control plane nodegroup and we cannot delete it due to the same issue as in 2.
Latest CAPI helm chart version causes autoscaling to stop working in a persistent hub setup, so we had to downgrade it to a previous version

2i2c-org/infrastructure#5601

Conclusion #

The biggest plus, is the people. We got support from Julian Pistorius, which has helped us a lot to both fix and validate some of the behaviours we were experiencing. Also, going through the Jetstream2 support process was also a pleasant experience because they were super prompt in answering and they were very nice.

Jetstream2 has a big plus over the other cloud providers with its openness thought the ACCESS program. This is something very handy to researchers and less costly than other cloud providers. 2i2c being able to offer hubs though this ACCESS program makes things more accessible to more researchers and more cost efficient.

Higher complexity comes also with more control over the infrastructure which has its advantages.

Leaving the challenges apart, the experience was a nice one and the outcome was positive -> 2i2c is now able to deploy both mybinder.org-like hubs as well as persistent storage hubs on Jetstream2 hardware, from the same cloud-agnostic infrastructure.

Acknowledgements #

Thanks to Project Pythia for funding and collaborating with us on this work.

Open infrastructure for collaborative geoscience with Project Pythia: Learning how to deploy a BinderHub on Jetstream2

Wed, 12 Feb 2025 00:00:00 +0000

Project Pythia and the “Jupyter notebook obsolescence” problem #

Project Pythia provides educational resources for essential software tools that enable open, reproducible and scalable geoscience, such as the Pangeo stack of packages (Xarray, Dask, Jupyter). Their Cookbooks are crowdsourced, community-curated, and open-source collections of Jupyter notebooks that demonstrate how to use these tools for cloud-native, geoscientific workflows (see our Project Pythia Cookoff blog post). However, “Jupyter notebook obsolescence” is a common problem: tutorials that were created a few years ago may no longer work due to changes in the software ecosystem and hampers the reproducibility of scientific results. A reproducible execution environment and the infrastructure to support it are essential for the long-term sustainability of these educational resources.

Leveraging NSF-funded cyberinfrastructure for BinderHub #

A BinderHub allows users to dynamically create custom computing environments from Binder-ready repositories containing computational notebooks and configuration files that describe the software environment required to run them. A public Binder service exists at mybinder.org (see our blog post about joining the mybinder federation 🎉) and is a successful example of how open cloud infrastructure can accommodate reproducible execution environments.

The resources available on such a public service are limited therefore 2i2c, together with Project Pythia, have been exploring how to deploy a BinderHub backed by larger resources from the NSF-funded cloud computing platform Jetstream2. This allows for larger simultaneous user loads, such as at workshops, as well as access to more powerful distributed and parallelized workflows required to process large geoscientific datasets, under a persistent resource allocation.

Learning how to deploy on OpenStack #

Jetstream2 uses OpenStack in order to manage pools of compute, storage and networking resources, and for our purposes we specifically make use of OpenStack Magnum Cluster API driver to manage Kubernetes for our deployment.

Cluster API needs a CAPI management cluster in order to manage other Kubernetes clusters, called workload clusters. On Jetstream2, this management cluster is gracefully created and operated by the Jetstream2 team, which means that the only task to worry about is creating and configuring the workload cluster.

For the workload cluster we used the Openstack Terraform provider to define the cluster template, the cluster itself and the node groups in a reproducible way.

After the cluster infrastructure was successfully created on Jetstream2, thanks to the 2i2c hub infrastructure being cloud agnostic as well, deploying BinderHub to Jetstream2, was a seamless experience and it was no different than on other cloud providers that we already supported.

We also learnt about some limitations of the Openstack Magnum driver project, which were expected given it being a relatively recent project, slowly being adopted, but they were all reported upstream and hopefully will soon be fixed.

Acknowledgements #

Jetstream2: Explore ACCESS allocation and Julian Pistorius for technical support
Thanks to Project Pythia for funding and collaborating with us on this work.
Andrea Zonca for preliminary work on Kubernetes deployments on Jetstream 2

2i2c hubs now run JupyterHub 5.0

Fri, 17 Jan 2025 00:00:00 +0000

We are excited to announce that all 2i2c hubs now run JupyterHub 5.0!

This is an upgrade that brings some exciting new features and improvements. Some of the highlights include:

The possibility to enable user-initiated server sharing
Authenticator-managed roles

Also, JupyterHub 5 will enable us to offer per-group shared directories in the future! Tracking Issue.

Checkout the JupyterHub 5.0 migration docs or the changelog for more details.

On the Jupyter Blog: From intern to mentor.

Fri, 30 Jun 2023 00:00:00 +0000

6——1———————– #

CILogon usage at 2i2c

Fri, 24 Feb 2023 00:00:00 +0000

About CILogon #

CILogon is an open source service provider that allows users to log in against over 4000 various identity providers, including campus identity providers. The available identity providers are members of InCommon, a federation of universities and other organizations that provide single sign-on access to various resources.

CILogon and 2i2c #

For the past year, 2i2c has been successfully using CILogon for more than fifteen of the hubs it manages.

Currently, most of the hubs that use it are hubs for communities in education that want to manage their hub access through their own institutional providers.

With using a tool like CILogon, we allow hub access to be managed both through the communities’ institutional providers, but also through social providers like GitHub and Google. Because both authentication mechanisms can coexist, there’s no need to provide specific credentials for 2i2c staff in order to have access to the hub. This reduces both the burden on institution’s IT departments, but also the complexity of a hub deployment.

Moreover, as we migrate away from our current Auth0 setup, the number of hubs using CILogon will further increase in the following year.

The setup #

The setup that 2i2c uses, is based on two important tools, the CILogon administrative client and the JupyterHub CILogonOAuthenticator.

The CILogon administrative client #

The 2i2c administrative client provided by CILogon allowed us to automatically manage the CILogon OAuth applications needed for authenticating into the hub.

For each hub that uses CILogon, we dynamically create an OAuth client application in CILogon and store the credentials safely, using the script at cilogon_app.py. The script can also used for updating the callback URLs of an existing OAuth application, deleting a CILogon OAuth application when a hub is removed or changes authentication methods, getting details about an existing OAuth application, getting all existing 2i2c CILogon OAuth applications.

The JupyterHub CILogonOAuthenticator #

For CILogon’s integration with JupyterHub’s authentication workflow, we’re using the CILogonOAuthenticator, which is part of the JupyterHub OAuthenticator project. This is what allows JupyterHub to use common OAuth providers for authentication, and it’s also a base for writing other Authenticators with any OAuth 2.0 provider.

As part of this 2i2c integration with the JupyterHub CILogonOAuthenticator some important upstream fixes and enhancements to the oauthenticator were identified and performed. For example, the GHSA-r7v4-jwx9-wx43 vulnerability was reported and fixed, and a migration guide containing a description of the breaking changes that were made, together with a step by step guide for the users on how to update their usage of JupyterHub CILogonOAuthenticator was provided.

Read more about how CILogon is setup for use at 2i2c from the docs.

Celebration #

Thanks to the 2i2c - CILogon partnership, during this past year we were able to integrate CILogon into 2i2c’s infrastructure and to observe its importance, usefulness and great support for 2i2c and the communities we server.

We are now happy to announce that the 2i2c - CILogon partnership has been expanded to another year!

Acknowledgements: The upstream jupyterhub-oauthenticator project mentioned in this post as being used at 2i2c is a JupyterHub package, kindly developed and maintained by the JupyterHub community and the 2i2c integration described was developed by the 2i2c engineering team. Also, this post was edited by Jim Basney.

Georgiana Dolocan

Mon, 01 Jan 0001 00:00:00 +0000

Software Engineer irreversibly in love with open source. A JupyterHub team member, focusing on infrastructure and community growth. Previously JupyterHub Contributor in Residence and Outreachy intern through an internship that supports diversity in open source and free software.