Impact report from 2i2c's Binder federation instance

Tue, 14 Oct 2025 00:00:00 +0000

The mybinder.org service provides reproducible and interactive computational environments for the open science community. It is financially supported by a federation of BinderHubs. 2i2c is part of the team that manages mybinder.org, and we’re committed to supporting this critical infrastructure for open science and reproducibility.

We developed a more cost-efficient process for deploying BinderHub on a single VM and are now running a BinderHub at 2i2c.mybinder.org. This post highlights the impact we’ve had through our support of the mybinder.org federation, we’ll update it periodically.

Usage over time #

Here are weekly launches on mybinder.org. Launches from 2i2c.mybinder.org are in red. Source: mybinder analytics dashboard.

In Q1 of 2025, 2i2c.mybinder.org launched 417,048 reproducible sessions. In this time, mybinder.org was primarily driven by our new Hetzner node as we worked with GESIS to stabilize their own BinderHub instance.

In Q2 of 2025, 2i2c.mybinder.org launched 249,750 reproducible sessions. In this time, we worked with GESIS to deploy their BinderHub instance on the same Hetzner node setup, which let us distribute more of Binder’s load onto them.

In Q3 of 2025, 2i2c.mybinder.org launched 118,083 reproducible sessions. We experienced a typical summer dip in sessions, had to fix the analytics dashboard as well as fix a TCP scanning abuse case that briefly brought down the 2i2c node.

Where we’ve made improvements #

As part of this effort, we’ve made several improvements to the Binder and JupyterHub ecosystem. Here are a few links where you can read more:

Deploying BinderHub on a single VM with k3s - Our approach to making BinderHub deployment cheaper and simpler
Hetzner cloud infrastructure experience - Cost-effective cloud hosting for mybinder.org
Combating TCP scanning abuse on mybinder.org - Developing anti-abuse tools to prevent an abuse use-case.
Improving Binder’s usage dashboard - This helps us create posts like these and is useful to others as well.
Integrating BinderHub with JupyterHub - Working with GESIS to bring Binder’s dynamic image building capabilities to persistent JupyterHubs, empowering users to manage their own environments

Acknowledgements #

This work is made possible by:

GESIS for their continued support as mybinder.org federation members
The JupyterHub community for collaboration and support
Our member communities whose fees support this work

Incident report: UC Merced user throttling during class startup

Tue, 16 Sep 2025 00:00:00 +0000

On August 29, 2025 our cloud infrastructure team experienced an incident with the UC Merced community hub when students tried to login simultaneously at the start of class. For more detailed technical information about this incident, see our full incident report.

What happened #

Students experienced issues when trying to login to the hub at the same time during the start of class.
The concurrent spawn limit was reached quickly due to the large number of users starting up simultaneously.
New nodes had to be brought up by the autoscaler, which took roughly 10 minutes from start to end.
Users who tried again after 1 minute weren’t guaranteed to get their servers started immediately since new nodes were still spinning up.
This was an “expected” scale-up event but the lack of clear messaging caused users to interpret it as instability.

What we learned #

We need better communication so users understand when infrastructure slowness is “expected” vs. “unstable”.
We need better alerting for concurrent user startup throttling - we found out about this issue from users rather than automated monitoring.
We learned that JupyterHub’s metrics don’t properly expose 429 status codes in our dashboards.
This will happen again if we don’t have proper scaling limits and node provisioning strategies for sudden user influxes.

Resolution #

We implemented several fixes:

Increased the concurrent spawn limit from 64 to 100.
Put UC Merced users on larger nodes to reduce the number of node spinups needed. this will cost more in cloud but result in fewer scale-up events.
Created action items to improve logging, alerting, and monitoring for similar incidents

Acknowledgements #

Thanks to UC Merced students and instructors for reporting the issue through our support system.

Cloud | 2i2c

Impact report from 2i2c's Binder federation instance

Usage over time #

Where we’ve made improvements #

Acknowledgements #

Incident report: UC Merced user throttling during class startup

What happened #

What we learned #

Resolution #

Acknowledgements #