Jenny Wong | 2i2c

Report from the Jupyter Security Working Group security tooling sprint

Wed, 08 Apr 2026 00:00:00 +0000

The Jupyter Security Working Group recently held a Security Tooling Sprint. It was a timely event given the recent spate of software supply chain attacks across the tech world.

The sprint covered two main areas:

Governance and strategy — conversations about responsibility and accountability in the face of AI, with emphasis on ensuring humans are ultimately responsible for code committed to Jupyter subprojects. The group also discussed how security could benefit from working group members regularly attending subproject meetings like the JupyterHub Collaboration Cafes.
Automation and tools — the group evaluated several tools for improving security posture across the Jupyter ecosystem. Here are a few that stood out:
- Semgrep as an alternative vulnerability scanner to CodeQL
- Grype, Checkov, and Kubescape for cloud infrastructure misconfiguration checks
- Schemathesis and restler-fuzzer for API fuzz testing

One challenge we discussed was how blindly running security scanning tools generates many false positives. There’s real effort needed to tune these tools for each project’s edge cases before they’re useful in automation. On a related note, we discussed the increase in AI-generated (or AI-assisted) vulnerability and security reports, and the challenges associated with sifting through all of those pieces of information.

Acknowledgements #

Thanks to the jupyter security working group for providing leadership and organizing, in particular Joe Lucas!
Thanks to the Jupyter Foundation for funding community meetings like these.

Better sharing UX with nbgitpuller and contextual error handling

Tue, 31 Mar 2026 00:00:00 +0000

TL;DR #

nbgitpuller now has improved UX context-aware error handling. Update to version 1.3.0 and let us know what you think by opening an issue or via the feedback form below 🚀

Check out the initiative we used to structure and complete this work:

GitHub initiative »

What is nbgitpuller? #

nbgitpuller is a way to sync content with compute through the click of a link. Example use cases include:

Interactive research demos, such as Spyglass (HHMI)
Workshops and training scenarios
University exams

In the case of Spyglass, the content is LorenFrankLab/spyglass-demo hosted on GitHub, the compute is a 2i2c cloud hub hosted at spyglass.hhmi.2i2c.cloud, and by using the handy nbgitpuller link generator, you can generate this nbgitpuller link to share with others to seamlessly explore content on the desired compute platform with the relevant data and toolchains installed.

How does nbgitpuller work? #

nbgitpuller is installed in the compute environment. The compute environment now has an /git-pull endpoint URL that can understand where to pull content from using URL parameters. Syncing content critically depends on git operations to fetch, checkout, clone, merge, commit, etc.

When it goes wrong #

Based on the data sent through by the kind folks running the Berkeley DataHub, there were 2163 logs available, of which:

983 (45%) were ‘merge’ conflicts
493 (23%) were ‘fetch’ errors
467 (22%) were ‘ls-remote’ errors

The remaining errors (10%) were mostly ‘checkout’ errors. This represents most frequent errors as seen by students.

In the same set of logs, there were 172 unique errors: most ‘ls-remote’ errors come from mistakes in the content repo URL. This represents most uniquely common errors made by instructors.

Merge conflicts #

If the link author changes content after the link consumer clicked a link, then nbgitpuller needs to sync updates for the consumer on subsequent link clicks. The nbgitpuller merging strategy makes opinionated choices so that the link consumer never has to interact with git, and will always preserve the consumer’s working changes.

Things can go wrong when

Consumers can diverge the git history if they perform a git commit
Authors can diverge the git history if they perform force push commits

Error UX (old) #

Problems with the old UX include:

a scary terminal
difficulty for the user to figure out what went wrong
no suggestion for the user to fix the problem or signpost to continue to the compute platform

Error UX (new) #

New improvements to the UX include:

terminal closed by default, but you can optionally toggle this open
there is a copy to clipboard button to easily share the error log from the terminal
more user-friendly and context aware helper message is displayed
a link to the general documentation for reference
a ‘Proceed without syncing’ button take the user to the compute platform without making any changes
in the case of merge errors an extra Backup and resync button option is presented

Give us feedback! Click here to provide feedback that will help us make this more impactful.

Learn more #

Acknowledgements #

UC Berkeley and the CloudBank Classroom project
CAL ICOR for co-funding
Eric Van Dusen and Sean Morris for championing this work
Balaji Alwar for providing the data and sharing feedback
Nicolas M. Thiéry for feedback on the UX design

Adding User Group Insights to Cloud Cost Dashboards with Grafana

Mon, 24 Nov 2025 00:00:00 +0000

We are excited to announce that we have extended our cloud cost dashboards to support display costs filtered by user groups using Grafana! This new feature allows administrators to monitor and manage cloud expenses based on user group memberships in JupyterHub.

Available for dedicated AWS clusters only (and excluding CloudBank managed accounts). Other deployments on GCP will be supported in the future.

Learn more #

Take a look at the Community Hub Guide to see what’s new
Check out the documentation of the 2i2c-org/jupyterhub-cost-monitoring project to see how it all works
Jenny recently presented her work on the cost monitoring system at JupyterCon 2025 earlier this month. Watch a video or look at the slides.

Give us feedback! Click here to provide feedback that will help us make this more impactful.

Acknowledgements #

Tarashish @ Development Seed for collaborating on this project with us.
NASA VEDA and the DSE Team at NASA MSFC ODSI for funding much of this work.
Kyle Lesinger from the NASA MSFC Office of Data Science and Informatics for providing valuable feedback and bug reports during development.

NASA Openscapes mentors run airborne data machine learning workshop with 401 participants from 68 countries

Thu, 23 Oct 2025 00:00:00 +0000

NASA Openscapes mentors recently hosted a workshop attended by 401 participants from 68 countries to learn how to use airborne data and machine learning for environmental research!

They gave participants computational environments on a community hub managed by 2i2c to teach skills in data science like image classification with airborne spectroscopy and accessing data from NASA JPL AVIRIS-NG data from Earthdata cloud.

We’re proud to enable participants from all over the world to easily access standardized compute and NASA Earthdata data in the cloud for a seamless learning experience.

Learn more #

Acknowledgements #

Thank you to the Openscapes team for sharing this post with us, and
- Michele Thornton
- Rupesh Shrestha
for authoring the blog post.
NASA Openscapes mentors for running a great workshop

Enabling transparent cloud cost monitoring with user-level dashboards

Tue, 30 Sep 2025 00:00:00 +0000

We are excited to announce that dashboards to monitor cloud usage and costs at a per-user level are now available! See the cost monitoring documentation for more information.

A key goal of 2i2c is to make the cloud safe for science. By providing transparent cost monitoring, we give communities the confidence that they won’t face unexpected bills and can better understand how their usage patterns translate to cloud costs. This visibility is especially valuable in our shared platform model, where each community gets their own independent hub while benefiting from shared infrastructure expertise.

The user-level cost breakdown allows communities to identify individual usage trends and manage their resources more effectively. Communities can now see exactly how their computational work translates to cloud spending, enabling better resource planning and budget management.

Give us feedback! Click here to provide feedback that will help us make this more impactful.

Learn more #

Cost monitoring documentation

Acknowledgements #

Tarashish @ Development Seed for working on this with us.
NASA VEDA for funding much of this work.
Andy @ Openscapes, Alex @ Development Seed and Sarah @ Earthscope for giving us close feedback.

Announcing `jupyterhub-groups-exporter`: monitor usage based on JupyterHub group membership with Prometheus and Grafana

Wed, 11 Jun 2025 00:00:00 +0000

Managing user groups in JupyterHub can be a challenging task, especially in environments with dynamic user bases and complex group structures. This post describes how we can leverage the latest group management features in JupyterHub, along with Prometheus and Grafana, to monitor group-level resource usage effectively.

⭐ Members of 2i2c’s community network can use this feature in their hubs by following our cost attribution documentation.

Motivation #

Hub admins have a strong impetus to monitor usage and costs by user groups because it allows them to advocate for better funding and cost recovery models based on data-driven insights. Group-level resource monitoring can help them to answer questions like:

How many people participated in our workshop group?
How much GPU compute is our power user group using?
Is our resource usage cost-effective for X group persona or Y group persona?

Current methods and workarounds include:

ring-fencing resources for specific user groups personas, e.g. creating a separate hub for a workshop group, or creating a separate Dask cluster for a power user group, which increases the admin burden of managing multiple hub instances
writing custom scripts to aggregate per user metrics, that are already available, into groups – which can be time-consuming and error-prone

JupyterHub and user groups #

Recent key developments upstream in JupyterHub for groups management, such as Authenticator managed group membership, makes this piece of work a prime and timely opportunity to be tackled. For more technical details of these upstream contributions, see GitHub PRs jupyterhub/oauthenticator#735 and jupyterhub/oauthenticator#498.

Users can access JupyterHub using a variety of authentication methods. Authentication providers like GitHub have built-in user management features that allow admins to create and manage user groups. These groups can then be configured in JupyterHub to authorize access to the hub, as well as control access to certain hardware profiles.

Following the key upstream contributions above, we can leverage Authenticator-managed group membership to automatically pass user group memberships from the authentication layer to JupyterHub itself. This allows us to capitalize on JupyterHub’s REST API to retrieve user group memberships from other services, such as exporting them as Prometheus metrics.

Exporting user group memberships to Prometheus #

The jupyterhub-groups-exporter project provides a service that integrates with JupyterHub to export user group memberships as Prometheus metrics. This component is readily deployable as part of any JupyterHub instance, such as a standalone deployment or a Zero to JupyterHub deployment on Kubernetes.

The exporter provides a Gauge metric called jupyterhub_user_group_info, which contain the following labels:

namespace – the Kubernetes namespace where the JupyterHub is deployed
usergroup – the name of the user group
username – the unescaped username of the user
username_escape – the escaped username

Escaped usernames are useful because Kubernetes pods have characterset limits for valid pod label names (this limit does not apply to pod annotations). Storing both types of usernames allows us to join escaped versions with their more human-readable unescaped usernames.

Exposing this metric as an endpoint for Prometheus to scrape allows us to query and join groups data with a range of usage metrics to gain powerful group-level insights. Here is an example PromQL query that retrieves the memory usage by user group:

sum(
 container_memory_working_set_bytes{name!="", pod=~"jupyter-.*", namespace=~"$hub_name"}
 * on (namespace, pod) group_left(annotation_hub_jupyter_org_username, usergroup)
 group(
 kube_pod_annotations{namespace=~"$hub_name", annotation_hub_jupyter_org_username=~".*", pod=~"jupyter-.*"}
 ) by (pod, namespace, annotation_hub_jupyter_org_username)
 * on (namespace, annotation_hub_jupyter_org_username) group_left(usergroup)
 group(
 label_replace(jupyterhub_user_group_info{namespace=~"$hub_name", username=~".*", usergroup=~"$user_group"},
 "annotation_hub_jupyter_org_username", "$1", "username", "(.+)")
 ) by (annotation_hub_jupyter_org_username, usergroup, namespace)
) by (usergroup, namespace)

Visualizing user group resource usage with Grafana #

The PromQL query above is rather long and complex to construct! However, you can benefit from an upstream contribution to the jupyterhub/grafana-dashboards project where we have encapsulated the PromQL queries as Jsonnet code and represented them as Grafana Dashboard visualizations (also known as Grafonnet). If you have a Kubernetes cluster running JupyterHub, try deploying these Grafana Dashboards and let us know what you think!

Our particular PromQL query above is visualized in the Grafana Dashboard User Groups Diagnostics under the Memory Usage panel (see also the corresponding screenshot at the top of this post). This is equivalent to its counterpart User Diagnostics dashboard, but with resource usage visualized on a per-group level rather than a per-user level 🎉

Future work #

We have laid the foundation for joining user group data to usage metrics with Prometheus by extracting memberships from JupyterHub’s database. This unlocks potent ways in which observability systems can be extended to group-level reporting and monitoring.

Future directions for this work include:

visualising cloud cost by user group in Grafana
developing more group-level reporting and monitoring dashboards
introducing group-level resource quotas.

What do you think? How would you like to see JupyterHub’s group management features evolve? Have you tried deploying this yourself? We welcome your feedback and feel free to open GitHub issues or make contributions to the repositories mentioned in this post.

Acknowledgements #

Thanks to the JupyterHub project for their collaboration and review of this work.

Announcing the Jupyter Book 2 alpha

Mon, 18 Nov 2024 00:00:00 +0000

Cross-posted from the Jupyter Book blog. Note that some MyST functionality is not supported on the 2i2c website – please see the original post for previews.

Over the last ten months, the Jupyter Book team have been hard at work; Jupyter Book has become a Jupyter subproject, and the team¹ announced a plan to release Jupyter Book 2. This post announces the alpha release of Jupyter Book 2.0, which has been re-written from the ground up to use the new MyST-MD engine.

Over the next few months, we will work in preparation for the full release of Jupyter Book 2. Stay tuned for more! The initial documentation for the alpha release can be found at:

next.jupyterbook.org/

Install the Jupyter Book 2 Alpha #

The Jupyter Book 2 alpha is available from PyPI.org. You can install it with pip, using

pip install -U jupyter-book==2.0.0a0

If you use pipx, it’s recommened to run Jupyter Book 2 using

pipx run jupyter-book==2.0.0a0

Jupyter Book 2 needs Node.js installed on your computer. If this is not the case, running jupyter book will prompt you to install it using the nodeenv package that ships with Jupyter Book 2:

❗ Node.js (node) is required to run Jupyter Book, but could not be found`.
❔ Install Node.js in '...'? (y/N):

Press y and Enter to proceed.

The Jupyter Book 2 project is a complete re-write of Jupyter Book. We expect there to be bugs and breakages! Please use our support channels to keep us up to date with your findings!

Discord

GitHub Issues

New Features in `2.0 alpha` #

Rich Hover Previews #

The new MyST book and article themes provide useful hover previews for links to other MyST content, Wikipedia, GitHub issues, and many more.

Content from other websites built with the MyST engine can be embedded in your own sites and PDFs:

Cross-referenced content can easily be embedded and re-captioned into other pages and projects, such as this figure to mystmd.org/guide/embed#mylabel.

Simple Instant Search #

A new client-side search uses a simple, modern algorithm for fast local search that finds the results that you care about.

Client-side search uses simple, modern, Algolia-inspired search algorithm to provide useful search results. We will be iterating on this in the near future for even richer search results!

High Quality PDFs #

PDF documents can now be built with Typst, a high-quality typesetting engine that produces readable error messages and beautiful documents. This feature was the basis for the 2024 SciPy proceedings, which is now built on MyST Markdown and will be accepting Jupyter Notebooks in 2025.

Example of the LaPreprint Typst template for rendering PDFs from Jupyter Book (via the MyST Engine).

Coming Soon in `2.0 beta` #

Custom Styles & Scrips #

Jupyter Book 2 will make it easy to tweak your website styles, and add new website behaviors.

Generate Markdown from Code Cells #

The MyST engine is on-track to support the inclusion of references and other markup features generated by code cells.

Control Cell Visibility with Tags #

In the beta release, Jupyter Book 2 will once-again be able to show and hide content according to cell tags.

Jupyter Book 2 vs MyST-MD #

At this early stage, the new Jupyter Book application jupyter book behaves identically to the mystmd engine that it is built upon; as outlined in our Jupyter Book 2 plan, we intend for Jupyter Book to be an “opinionated distribution” of mystmd that shares the same configuration format and CLI. This contrasts with Jupyter Book 1, which was built on top of the Sphinx documentation engine, but offered its own CLI and configuration files. In future, the jupyter book and mystmd CLIs may diverge from one another, but we expect that this will be handled in a graceful manner: mystmd commands should always be compatible with the jupyter book application.

Acknowledgements #

Thanks to Project Pythia for funding some of our ongoing work on Jupyter Book.
Thanks to the Jupyter Book community for their collaboration.

Jupyter Book project has historically been a technical project of the Executable Books organisation. In 2024, the establishment of a Jupyter subproject means that the Jupyter Book project now has its own identity outside of Executable Books. ↩︎

Track and manage cloud costs using Grafana

Fri, 15 Nov 2024 00:00:00 +0000

Grafana dashboard showing cloud costs broken down by compute, storage and other components for the Openscapes hub.

We are pleased to unveil a new feature to track cloud costs within our Grafana dashboards! Community Champions now have the ability to monitor the cost and usage of their 2i2c-managed hubs that displays up to date aggregated costs as well as detailed breakdowns for more granular reports.

Note that this feature is currently available to AWS hosted hubs only and will be rolled out to other cloud providers in the future.

Accessing the cloud cost dashboard #

Community Champions can view the Cloud Cost dashboard from their Grafana instance (please see the Service Guide for how to gain access).

From the main menu of Grafana, navigate to Dashboards > Cloud cost dashboards > Cloud cost attribution to view the dashboard.

Understanding the cloud cost dashboard #

A typical 2i2c-managed deployment comprises of a staging hub and a production hub, although some other communities may have extra hubs such as a workshop hub. By default, costs are not broken down on a per hub basis unless the community has opted in to this feature.

The dashboard is made of several panels:

Daily costs
Daily costs per hub (opt-in only)
Total daily costs per component
Daily costs per component per hub (opt-in only).

For more detailed information on the data that each panel displays, please consult our Service Guide for reference.

The dashboard can be shared with other community members and stakeholders so they can understand usage and cost patterns. Community Champions can export data to a CSV file, or they can generate a snapshot of the Grafana dashboard and share a public link.

For instructions on how to export data from the dashboard, please see our Service Guide for reference.

Next steps #

We would love to know whether this feature is useful and how it can be improved. We will be contacting individual communities to share their feedback with us – please share your thoughts with us!

We will work on rolling out this service to GCP hosted clusters in future. Stay tuned to know when this feature is available to your community.

Acknowledgements #

Thank you to Erik for spearheading the rollout effort and to the rest of the 2i2c team for their support.
Thanks to Openscapes and Cryocloud communities for providing valuable insights during the prototyping and testing phase, and for funding part of this work.

Introducing the Catalyst Project Community Partner Highlights

Thu, 31 Oct 2024 00:00:00 +0000

Cross-posted from the Catalyst Project blog

Catalyst Project community partners using accessible cloud infrastructure for open science leadership and training. (clockwise from top-left) NNB-CCG, MUST, CICADA and INER. Photos courtesy of Shirley Alquicira Hernández, Bennett Kankuzi, María Inés Fariello Rico and Yalbi I. Balderas-Martinez.

The Catalyst Project is a community-engaged initiative designed to support the adoption of open science principles in under-served bioscientific research communities through the provision of reliable and sustainable cloud computing infrastructure. It’s a project we’ve been working on now for almost two years, which involves staff from seven different organizations: 2i2c, The Carpentries, CCAD, CSCCE, IOI, MetaDocencia, and OLS, and is funded by the Chan Zuckerberg Initiative.

A key part of the project is engaging with Community Partners in Africa and Latin America: Institutions, organizations, and individuals who are undertaking bioscientific research projects that require cloud computing infrastructure. As collaborators on the Catalyst Project, Community Partners can access and use 2i2c’s open science cloud services, and also receive training from 2i2c, The Carpentries, MetaDocencia, and OLS to support their work. Community Partners also play a vital role in shaping an evolving governance model for the Catalyst Project to help sustain, scale, and maximize impact in Latin America, Africa, and under-served communities around the world.

In a new collection of blog posts (that we hope will expand over the next couple of months!) we’re highlighting the work of the Catalyst Project Community Partners. This post is a gateway to learning more about the Catalyst Project and its Community Partners. If you have any questions or feedback about the project, please send an email to the core team.

Highlighting the Catalyst Project Community Partners #

The Catalyst Project currently involves 19 Community Partners, 9 in Africa and 10 in Latin America. Our initial blog post series showcases seven of the Partners, and each post is available in English and Spanish:

African Institute of Biomedical Science and Technology (AiBST) #

At the African Institute of Biomedical Sciences and Technology (AiBST) in Zimbabwe, Zedias Chikwambi and colleagues are working to discover and utilise biomarkers to guide personalized medical treatment.

“Through the Catalyst Project we are able to bring genomic information interpretation to patient bedsides.” - Zedias Chikwambi

Read more: EN | ES

Malawi University of Science and Technology (MUST) #

The Catalyst Project Community at the Malawi University of Science and Technology (MUST) is working to popularize the applicability of data science and artificial intelligence in various undergraduate and postgraduate research contexts in Malawi, including health, business, and education.

“…many staff and students need… a robust and easily accessible platform from which they can efficiently run their machine learning models and do advanced data analysis for their data science research…” - Bennett Kankuzi

Read more: EN | ES

MolerHealth #

MolerHealth is focused on revolutionizing healthcare in Nigeria by developing an accessible electronic health record (EHR) system aimed at reducing disease misdiagnosis.

“Access to training, like The Carpentries Instructor Training, has empowered our team with essential skills for effective teaching and collaboration.” - Monsurat Onabajo

Read more: EN | ES

Nelson Mandela African Institution of Science and Technology (NM-AIST) #

The Northern Tanzania One Health Research Group, hosted at the Nelson Mandela African Institution of Science and Technology (NM-AIST), is using the Catalyst Cloud Infrastructure to understand the transmission dynamics, genetic diversity, and antimicrobial resistance of the Mycobacterium tuberculosis complex (Mtb) between humans and livestock in Northern Tanzania.

“…access to training, particularly through the 2i2c Hub Champion Training, has significantly enhanced our ability to manage and optimize cloud-based resources." - Beatus M Lyimo

Read more: EN | ES

Nodo Nacional de Bioinformática (NNB-CCG) #

The Nodo Nacional de Bioinformática (NNB-CCG) of the Centro de Ciencias Genómicas (CCG) - Universidad Nacional Autónoma de México (UNAM) is a group that brings together professionals and academics to support, provide services, and maintain the growth of the field of bioinformatics in the country’s research.

“(Our) goal is to optimize our participation in events, assess the usefulness of the Catalyst Project’s resources, and, in turn, provide the Catalyst Project with guidelines to improve their service by identifying the necessary areas for improvement within the institutions." - Shirley Alquicira Hernández

Read more: EN | ES

Instituto Nacional de Enfermedades Respiratorias (INER) #

Collaborators at INER are using the Catalyst Cloud Infrastructure to implement machine learning algorithms that will classify radiology images of pulmonary thromboembolism (PTE), with a view to improving patient outcomes.

“The Catalyst Project is helping us to collaborate more efficiently and work remotely.” - Yalbi I. Balderas-Martinez

Read more: EN | ES

Centro Interdisciplinario en Ciencia de Datos y Aprendizaje Automático (CICADA) #

CICADA, an interdisciplinary center researching data science and machine learning, is using the Catalyst CLoud Infrastructure to analyze Uruguayan population data to understand patterns of migration, how much of the native footprint remains, and what can be said about the people who lived in the Uruguayan territory before the arrival of Europeans.

“The Catalyst Project…trainings are attractive, as they are respectful of the people, no previous knowledge is assumed, and the instructors are welcoming.” - María Inés Fariello Rico”

Read more: EN | ES

About the blog posts – Acknowledgements #

Creating a series of blog posts to highlight the work of the Catalyst Community Partners was a collaborative effort involving staff from CSCCE, 2i2c, OLS, and MetaDocencia. Specifically, Lou Woodley, Katie Pratt, Jenny Wong, and Tajuddeen Gwadabe conceived the idea during our regular Catalyst Project “website team” meetings, and developed a strategy for reaching out to community partners to gather information.

Katie then led that outreach, which involved a representative of each Community Partners completing a Google Form to answer three key questions and sharing a photo (or a selection of photos) to use on the blog. Katie curated all of the information, edited together the blog posts, and secured sign-off from all of the partners. Sabrina López and her team at MetaDocencia coordinated the translation of the posts from English to Spanish, and Jenny formatted and published them on the website.

Acknowledgements #

Thanks to the Catalyst Community Partners for generously sharing more about their work.
Thanks to CZI for funding this work.

MyST Mini-Hackathon with the DeepLabCut Team

Mon, 02 Sep 2024 00:00:00 +0000

The DeepLabCut Team #

Animal pose estimation using deep neural networks. Courtesy of the DeepLabCut Jupyter Book

The DeepLabCut team is a group of researchers and developers who are working on open source tools for analyzing animal pose estimation by training deep neural networks on videos.

Chris Holdgraf visited the lab in early August to learn more about how the group were using open-source tools to document and share their work.

Jupyter Book and MyST #

Extensive documentation for using the DeepLabCut software package is already available as a Jupyter Book. The group was interested in adopting MyST Markdown to stay ahead of the curve and upgrade their Jupyter Book (see the related announcement Jupyter Book 2 will be build upon the MyST-MD engine).

Chris led a mini-hackathon to introduce the group to MyST and collect feedback on where enhancement features could be made in the future. Here’s a summary of the outcomes:

Many improvements were made to the MyST documentation 📖
- The MyST Quick Start Guide was used to onboard new users. Amendments were upstreamed to the MyST docs directly and were immediately available to all.
- A tutorial on executable documents was added to the collection of MyST tutorials.
- MyST-MD installation instructions were simplified using mamba.
A bunch of enhancement features were requested ✨
And we found a bug in the table of contents validation 🐞

Summary #

Hackathons are a great way for quickly imparting knowledge and gathering feedback in a short space of time. The event spurred rapid contributions to the MyST ecosystem – embracing reuse of the MyST quick start guides saved time and effort, while engaging with users directly closed a tight feedback loop for enhancements.

Acknowledgments #

Thank to the Mackenzie Mathis Lab for hosting Chris Holdgraf at EPFL, Lausanne, Switzerland.
Thanks to the Jupyter Book team for collaborating on this with us.

NeuroHackademy Summer School Reflections

Mon, 19 Aug 2024 00:00:00 +0000

Thank you to Ariel Rokem and Noah Benson for guest writing this blog post!

Group photo from NeuroHackademy 2024

What is NeuroHackademy? #

Part summer school, part free-wheeling hackathon, all focused on the use of data science methods in neuroscience, NeuroHackademy is an event that was recently hosted by the University of Washington eScience Institute in Seattle, WA, USA. This event, that has been running annually since 2016, aims to provide early-career researchers in Psychology, Medicine, Neuroscience, and other related fields with the skills and knowledge that they need to effectively and rigorously work with open source tools and workflows for analyzing human neuroscience data. This supports the effort to make scientific analysis and results shareable, reproducible, and accessible.

Global and inclusive #

In 2020, the event had to rapidly pivot to an online format, and this format was also used in 2021. Through this experience, the organizers ( Ariel Rokem and Noah Benson) realized that many participants preferred the online format. For example, participants who could not afford to travel to Seattle, or participants who had care-taking responsibilities that precluded them from participating in a two-week event away from their homes. In 2022, the event pioneered a hybrid format, where half of the participants are present in-person and half join the event via zoom, slack, GitHub, and of course through a dedicated 2i2c JupyterHub. Taken together, this format allows the participation of students from a larger range of backgrounds and locations. This aspect plays an important part in building a global and inclusive community of practice. See the paper Hands-On Neuroinformatics Education at the Crossroads of Online and In-Person: Lessons Learned from NeuroHackademy to read more on this subject.

Collaboration with 2i2c #

Previous years #

NeuroHackademy has been an early adopter of the cloud-based JupyterHub model, setting up its first hub using the zero-to-jupyterhub guide in 2018. NeuroHackademy partnered with 2i2c as soon as it was founded, and 2i2c has operated a JupyterHub for the last 3 years. The hub provides an interactive computing platform for learners, and implements the “digital watering hole” for practical and immediate access to a range of cloud-based datasets in human neuroscience (see blog post announcing support for this year’s event).

In terms of the software environment, the following tools and features that have benefited the event over the years include

nbgitpuller allows students to synchronise lesson content with an organizational GitHub repository that is collaboratively maintained by the lesson instructors.
Shared data file storage with read-only access for learners and read-write access for instructors
Access to an abundance of neuroimaging data hosted in cloud object storage
- The Human Connectome Project
- The Natural Scenes Dataset
- OpenNeuro
- The Healthy Brain Network
- And more.

This year #

This year 2i2c supported the following tools and features for NeuroHackademy

A “Bring your own image” option where users can pull any image hosted on a container registry into the hub. See our Integrating BinderHub with JupyterHub: Empowering users to manage their own environments blog post for more details.
repo2docker and GitHub actions to build and prototype images from a repository.
The support services provided by 2i2c and the ability for instructors to open pull requests on 2i2c infrastructure for speedy resolution.
GPU instances to support more compute intensive workloads for machine learning.

Next year #

One thing we have learned is that 2i2c automatically shuts down a user server after one hour of inactivity by default to ensure efficient resource usage and limit runaway cloud costs. Naturally, we are seeing increasing demand from learners for longer and more complex analyses. In response to this, we are keen to explore how the jupyter-keepalive extension can keep the server alive for long-running processes.

We are pleased that learners have made great progress in making use of cloud-native, open-source workflows for analyzing human neuroscience data. We are keen to benefit from lessons learned this year and are looking forward to collaborating with 2i2c once again to deliver the NeuroHackademy Summer School in 2025.

Watch this space next year!

Acknowledgements #

Funded by grant R25MH112480 from the US National Institute of Mental Health awarded to Ariel Rokem and Noah Benson.

The NeuroHackademy Summer School is sponsored by

References #

Keeping PACE with GPU enabled compute to detect global cloud cover using satellite data

Mon, 12 Aug 2024 00:00:00 +0000

(left, b&w) Model inputs and (right, color) model outputs of a simple multi-layer perceptron for detecting cloud cover.

PACE is the NASA Plankton, Aerosol, Cloud, ocean Ecosystem mission that focuses on understanding ocean health and its impact on the atmosphere. Together with the Ocean Carbon and Biochemistry (OCB) program, a one-week hackathon ran from Aug 4 to Aug 8 on the 2i2c-hosted CryoCloud hub. The goal of the hackathon was to explore new Earth science data streams provided by the OCI, SPEXone and HARP2 instruments using Python.

Machine Learning with GPUs #

One of the most advanced tutorials delivered during the hackathon was the Machine Learning Tutorial. The tutorial focused on creating a machine learning pipeline to detect cloud cover from satellite imagery. This was done by training a convolutional neural network (CNN) to assign each pixel a binary value to indicate whether the location was covered by cloud or not. To improve the spatial context beyond a single pixel value, as the likelihood of a pixel containing cloud cover increases if its neighbours also contain cloud cover, the CNN needs to be trained on the entire image at once rather than at a single pixel level. This massively increases the training time, but also allows the CNN to learn more complex relationships between pixels.

GPUs have a far greater number of cores than CPUs that are well-suited for accelerating the massive parallel processing needed to train a neural network on the large amounts of image data in the above scenario. PyTorch is a popular Python library for training CNNs, available for both CPUs and GPUs, and is an ideal tool for performing this kind of work. In terms of the accelerator hardware available on the CryoCloud hub, 2i2c provisions an instance with an NVIDIA Tesla T4 GPU with 4 CPUS, 16GB of RAM and 2,560 CUDA cores.

Managing shared memory on 2i2c hubs #

While developing the above tutorial, tutorial lead Sean Foley (NASA/GSFC/SED & Morgan State University & GESTAR II) noticed that training neural networks was way slower than it should be given the GPUs available to them. They investigated the issue, and with help from the 2i2c engineering team, it was determined that shared memory was the issue. PyTorch uses shared memory via /dev/shm for faster parallel processing, and maximizing use of GPU. However in containerized environments, this is limited to a maximum of 64MB by default.

You can check the amount of shared memory available on your hub in a terminal with the command

df -h | grep /dev/shm

As you might expect, 64 MB of shared memory is not enough for training over 160,000 images in the tutorial. 2i2c was able to remove the limit, making /dev/shm share the memory the user has selected via their profile list, rather than be artificially limited to any particular size. This was done for all users on the CryoCloud hub within an hour of the issue being reported and we upstreamed the change for all 2i2c hubs (see GitHub pull requests for CryoCloud and all 2i2c hubs).

Conclusion #

This event demonstrates the economy of how running shared and open infrastructure dynamically solves problems for the benefit of many users, not just for one occasion. Learning experiences such as the above are transferred and embedded upstream into transparent and flexible open source software that impacts not only all users of 2i2c operated hubs, but also generalized for the wider research community at large (case in point, see the Slack thread below from Eli Holmes, operator of the NOAA Fisheries hubs)! We are grateful for the strong partnerships with our communities who help us to co-design impactful solutions that are specific for their needs and accessible to all.

The power of open infrastructure beyond 2i2c-operated hubs

References and Acknowledgments #

Sean Foley (NASA/GSFC/SED & Morgan State University & GESTAR II)
Tasha Snow (ESSIC UMD & NASA GSFC & CryoCloud)
PACE Hackweek Jupyter Book

Collaborating with Development Seed to deliver cyberinfrastructure for NASA VEDA

Fri, 12 Jul 2024 00:00:00 +0000

Thank you to Sajjad Anwar and Sanjay Bhangar for contributing to this post.

The VEDA dashboard

The 2i2c team are proud to continue our strong working collaboration with Development Seed, following our previous work on launching the US GHG center (also see the Development Seed blog post). Together with scientists at NASA in our regular sync touchpoints, we have recently delivered a tranche of improvements to the Visualization, Exploration and Data Analysis (VEDA) project.

This platform is designed to thread open-source components together to consolidate GIS delivery mechanisms, processing, analysis and visualization tools, and presented in a collaborative interactive computing environment. All code repositories and associated resources stemming from this work are available on the VEDA GitHub page.

In the spirit of fully open development, you can see the objectives the combined 2i2c and Development Seed team had for the last quarter. In this blog post, we will describe some of the significant ones!

Better image management and testing #

The repo2docker-action is a GitHub action simplifying image building and testing for use with JupyterHub, using either a Dockerfile or various configuration files (like requirements.txt, environment.yml, etc) supported by repo2docker. We migrated our image building pipeline from a somewhat homegrown solution to this upstream action, making image updates and testing much easier. In particular, we can automatically run test notebooks on every change we make to the image! This way, we can easily catch any breaking changes in library versions or other package installs without disrupting users. We also debugged and contributed upstream fixes to the testing infrastructure so everyone could benefit from this, rather than just us.

Automatically pulling example notebooks on startup #

When a user logs into a JupyterHub, it is very helpful if we could have a bunch of example notebooks and other content pre-populated for them so they can get started right away. nbgitpuller is heavily used for this particular use case. However, it requires that nbgitpuller is installed inside the image the user is using - and not all images have it installed. In particular, we wanted to continue using the (wonderful) Rocker images maintained upstream for R users, however they do not have nbgitpuller installed. To solve this problem we built jupyterhub-gitpuller-init, which can be used as an init container to pre-populate user content on persistent home directories regardless of the image used. We also made sure to build this in a way that anyone can use it, and it is not tied into either 2i2c or VEDA infrastructure!

Opening specific visualizations in QGIS via URL #

QGIS is the world’s most used open source GIS software, and previously 2i2c had worked with Openscapes and QGreenland to bring this desktop software to JupyterHub. We had previously worked on a container image that allows users to access large datasets stored in the cloud directly through QGIS on the JupyterHub, allowing users to work with much larger datasets than they could on their desktops by bringing cloud compute adjacent to the data. As a continuation of this work, we developed jupyter-remote-qgis-proxy, which builds QGIS specific features on top of jupyter-remote-desktop-proxy. In particular, it allows creation of shareable links that when clicked, opens specific datasets and layers in QGIS in a JupyterHub! You can see this in action:

Launching QGIS on a Linux desktop served by the VEDA JupyterHub

This opens up exciting future possibilities. Imagine this exploration of the Camp Fire having an ‘Open in QGIS’ button that enables further exploration of the data without the user needing to download or install anything! Work will continue in the coming quarter towards achieving this vision.

We are also excited to see recent work in this space from QuantStack and Simula Labs, and will follow up to ensure an orderly transition to more web native workflows for existing users of QGIS in due time.

Better Profile Selection #

This is a continuation of our GESIS collaboration. In the path to deploying dynamic image building to end users, we wanted to stabilize jupyterhub-fancy-profiles enough to deploy to users of VEDA (and eventually everyone else). This is the primary interface users see after they log in to JupyterHub, and was ripe for UX improvements. The default interface looks like this:

The revamped one is much more streamlined and looks like this:

Revamped Profile Screen

This is currently deployed to a staging hub and has helped us shake out a lot of bugs! We expect the improved interface will be rolled out to all users in the near future. We are also planning further development to make the user experience even better and smoother for everyone.

Supporting workshops #

End users benefiting from our work is what ultimately gives meaning to our work. To that end, we were very happy to support running workshops during this collaboration – see our related blog post US Greenhouse Gas Center supports summer school at CIRA for more information.

Ongoing Collaboration #

Delivering on these objectives in a timely way heavily depended on the success of the team collaboration. Sanjay Bhangar of Development Seed commented

Working closely with the 2i2c team on growing features to support users on the VEDA and GHG Center hubs has been absolutely amazing. With 2i2c’s deep experience in the Jupyter ecosystem, we have been able to implement some fairly complex features quite easily, and their strong open-source roots have ensured that whatever we work on is broadly useful to the wider Jupyter and scientific computing communities.

Take a look at the companion Development Seed blog post of this work.

This collaboration continues, and we have now published our objectives for the coming quarter. Watch this space!

Acknowledgements #

Development Seed
NASA IMPACT
Tarashish Mishra, Julia Signell, Oliver Roick, Slesa Adhikari and Sanjay Bhangar for various code contributions towards these objectives

Openscapes Host a Surface Biology and Geology Workshop with Shared Password Feature

Tue, 09 Jul 2024 00:00:00 +0000

Thanks to Brianna Lind, Julia Lowndes and Andy Teucher for contributing to this blog post!

Surface Biology and Geology: VITALS Workshop

Openscapes is a value-based initiative that supports kinder, better science based on open source community. NASA Openscapes is in its fourth year as a project supporting NASA Earth science in the Cloud, co-developed by Julia Lowndes (Openscapes) and Erin Robinson (Metadata Game Changers).

The initiative recently supported the Surface Biology and Geology: VITALS Workshop hosted by NASA Land Processes Distributed Activate Archive Center (LP DAAC) and NASA Jet Propulsion Laboratory (JPL).

Instructors used the 2i2c Openscapes Hub to lead hands-on exercises teaching learners how to manipulate data collected from the ECOSTRESS and EMIT instruments onboard the International Space Station. They used Jupyter notebooks in the Hub to demonstrate how open source tools together with cloud data and compute resources could effectively analyse the the Canopy Water Content and the Land Surface Temperature over the Jack and Laura Dangermond Preserve, Santa Barbara, CA.

Plot of the Canopy Water Content over the Jack and Laura Dangermond Preserve, Santa Barbara, CA from a VITALS Workshop Jupyter notebook.

This event was attended by around 250 participants. An event of this size therefore requires a frictionless login flow so that organizers could focus on the essential complexity of teaching data analysis rather than the accidental complexity of managing Hub authorization. GitHub authentication is the default option for most 2i2c Hubs for research use cases, but for an educational event of this size this option was not fit for purpose since organizers had to

Retrieve the GitHub usernames of each participant (assuming everyone was familiar with GitHub!)
Manually invite GitHub users to a GitHub organization to authorize access to the Hub (invitations would expire within seven days)
Repeat the above two steps last-minute for participants who showed up on the day without preparing
Manually remove GitHub users from the GitHub organization if they wanted to revoke access to the Hub after the event.

In response to this need, we developed a shared password feature so that workshop organizers can simply hand the share password out to learners for access to the Hub. This bypassed the manual labour of managing GitHub accounts while not adding to the learner’s high cognitive load and improving the participant’s learning experience overall.

One of the elements that enabled us to recognize and solve this issue effectively is our close partnership with the Openscapes team. We engage in regular 6-weekly catch-ups where we can learn about user requirements and how we can develop our infrastructure to co-create optimal solutions. Together with our Product Delivery Flow, we were quickly able to architect the shared password solution in time for the workshop.

Feedback from Brianna Lind (LP DAAC)

We have documented the technical infrastructure changes required to enable a shared password for the Hub in our Infrastructure Guide and hope to support many future events with this mechanism!

Acknowledgements #

NASA Openscapes
NASA LP DAAC
NASA JPL
NASA ROSES funding
NASA Open Science / ScienceCore for supporting some of our work on JupyterHub.

Determining Climate Risks with NASA Earthdata Cloud at Scipy 2024

Mon, 08 Jul 2024 00:00:00 +0000

Determining Climate Risks with NASA Earthdata Cloud is a ScienceCore curriculum module that comprises part of NASA’s Open Science and Transform to Open Science (TOPS) initiatives. The aim of this module is to deliver a hands-on experience with “data-proximate computing” in the cloud with NASA Earthdata products with content co-developed with MetaDocencia.

This module was delivered as a SciPy tutorial at this year’s conference. 2i2c have been working closely with the organizers to provide the hub infrastructure for the tutorial, including enabling a shared password for easy authentication (see our Openscapes post for more about this feature) and operating a small binder service for participants to view content after the event.

You can take a look at the tutorial on the NASA ephemeral hub!

The event was well-attended, with 40 learners taking part. Special thanks go to the organizers Dhavide Aruliah, Karthik Venkataramani and Patricia A. Loto for leading the tutorial.

Acknowledgements #

NASA F.14 Transform to Open Science Training award NNH22ZDA001N-TOPST (80NSSC23K0861), which seeded this contribution to NASA ScienceCore
MetaDocencia
Dhavide Aruliah
Karthik Venkataramani
Patricia A. Loto

Enabling neuroscience in the cloud with HHMI Spyglass and MySQL on JupyterHub

Fri, 05 Jul 2024 00:00:00 +0000

The HHMI Spyglass tutorial

Spyglass #

Spyglass is a framework for reproducible and shareable neuroscience research produced by Loren Frank’s lab at the University of California, San Francisco. Check out our blog post about the release of their preprint to read more about the methods.

This post focuses on the complex data storage needed for the project, which can be difficult to set up locally or at scale in the cloud. In particular, the analysis needed a MySQL database for reproducibility. This is a fairly common task across many fields. The aim of 2i2c is to enable researchers to focus on the essential complexity of what they were doing, i.e. the science, without managing the accidental complexity of how to do it – in this case, setting up databases.

We describe how you can do this too for your own JupyterHubs. Since 2i2c commits to running our infrastructure in line with open-source values as much as possible, you can also directly see the configuration for the hub referenced in the paper.

What is a “sidecar container”? #

The Kubernetes definition of a sidecar container is

Sidecar containers are the secondary containers that run along with the main application container within the same Pod. These containers are used to enhance or to extend the functionality of the primary app container by providing additional services, or functionality such as logging, monitoring, security, or data synchronization, without directly altering the primary application code.

In this case, the primary app container is the JupyterLab instance where people are interactively running code and doing science. We want to provide a MySQL database as a sidecar so that each user server gets their own independent MySQL server instance (that is not accessible to anyone else). We can then run code such as

%%bash
mysql -h 127.0.0.1 -u root --password=tutorial < path-to-sql-file-with-data

to load data into the database. Note the IP address 127.0.0.1 - the MySQL server is listening on localhost, even though it is not running in the same container! Thanks to the magic of Linux Network Namespaces, the sidecar and main app container can share 127.0.0.1. This allows you to write code that works in the exact same way on a user’s local computers as on the JupyterHub, making transitions and replication easier.

Setting up sidecars in JupyterHub on Kubernetes #

We’re leveraging multiple tools from the open-source ecosystem - JupyterHub, Kubernetes, Linux as well as MySQL itself.

Since this is a Kubernetes feature, we can pass through config to it. There are two layers here, which are

singleuser.extraContainers in z2jh configuration
KubeSpawner.extra_containers in KubeSpawner configuration

The hub configuration looks like

 singleuser:
 extraContainers:
 - name: mysql
 image: datajoint/mysql:8.0 # following the spyglass tutorial at https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/#existing-database
 ports:
 - name: mysql
 containerPort: 3306
 resources:
 limits:
 # Best effort only. No more than 1 CPU, and if mysql uses more than 4G, restart it
 memory: 4Gi
 cpu: 1.0
 requests:
 # If we don't set requests, k8s sets requests == limits!
 # So we set something tiny
 memory: 64Mi
 cpu: 0.01
 env:
 # Configured using the env vars documented in https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/#existing-database
 - name: MYSQL_ROOT_PASSWORD
 value: "tutorial"

By setting this up, we allow users to insert the code snippet above

%%bash
mysql -h 127.0.0.1 -u root --password=tutorial < path-to-sql-file-with-data

into their Jupyter Notebooks, which gives access to their MySQL database in the hub!

However, this configuration does not include permanently store the database itself between hub server sessions. Thanks to a pilot in a prior collaboration with University of Texas, Austin, we do have some documentation on how you can enable that as well!

Acknowledgements #

Howard Hughes Medical Institute
National Institute of Mental Health (NIMH), grant number RF1MH130623
kubespawner
zero-to-jupyterhub-k8s and the JupyterHub community

We need to improve the diversity of our leadership

Fri, 28 Jun 2024 00:00:00 +0000

As an organization committed to broadening access to interactive computing for global communities, we believe that a team embedded with diverse insights and lived experiences can more effectively advocate for underrepresented voices in our socio-technical partnerships. A diversity of team experiences helps us deliver a service that broadens and empowers participation in open-source science for all communities and leads to more effective teams that make better decisions.

As a young and growing organization, we have had mixed success in building diversity within our team. We’ve leaned into the values of our service that build trust and attract a broad coalition of community partners (and potential employees). For example, we’ve used the following pillars in describing our service:

Transparency – our transparent and participatory model keeps our incentives aligned with community needs.
Empowerment – our service gives communities agency to design the service they need, and to manage it without 2i2c if they wish.
Partnership – our communities are partners working towards shared impact, not customers to grow revenue.
Sustainability – our service should have a self-sustaining model that ensures continuity, growth, and funder independence.

We’ve also invested in creating an inclusive organizational culture via documents like our Team Compass and our Code of Conduct¹. We’ve experimented with inclusive hiring practices to encourage a diverse pool of applicants for our open positions, such as running open office hours for job postings in the hopes that this would encourage more people to apply who might otherwise have been hesitant (something that often correlates with people from historically marginalized communities).

The importance of diverse leadership #

However, over the past few months, we have reflected on our organization’s structure and future plans. In particular, we recognize that there is a worrying lack of women in staff leadership roles. This imbalance does not live up to our values of diversity in our team and service.

We need to do better.

2i2c is at a moment of maturation and growth as an organization, kicked off by our organizational audit from 2023 as well as our three-year retrospective. We believe that improving the diversity of leadership throughout the organization is a necessary part of that maturation over the next three years.

Below are a few ideas for how we aim to make improvements, and we invite feedback from others who are interested in helping us improve this aspect of our organization.

What we plan to do #

Over the next few months, we will work on a long-term DEI strategy in tandem with our sustainability strategy as we work towards scalable and sustainable networks of community hubs.

Here are some steps we aim to take to improve the diversity of our leadership across the organization:

Define our goals for diversity. First, we need a better understanding of the most important axes of diversity that we wish to design around. Gender is clearly a critical gap to cover, but there are many other important axes of diversity as well. For example, as an organization that serves a global community, it is important that we have global perspectives represented in our leadership.
Define mechanisms for leadership and governance that includes representation along these axes. Leadership at a staff level is one way to ensure representation of diverse perspectives, but there are many other ways to bring voices into the conversation. We aim to explore new ways of bringing diverse voices into the strategic direction of the organization and provide mechanisms for holding us accountable to our values. For example, we intend to grow a board to guide 2i2c’s strategy and hold its Executive Director accountable for achieving impact - this is another opportunity to bring diverse perspectives into our leadership.
Make a plan for improving leadership diversity within our team. Whatever mechanisms we create for diverse leadership, we know that one of them needs to be improving our diversity at a staff leadership level. Our staff are the ones that spend the most time working on - and have the most leverage over - our strategy and mission, and it’s important that our staff leadership is a good representation of the diverse perspectives of the communities we serve. This might mean investing more heavily in reaching broad applicant pools for new positions, seeking external consultation for how we can avoid unintentionally sending exclusionary signals to others in our communications and outreach and developing incentives to retain our talent over time.
Continue improving our systems of delivery and work. Finally, we believe that a crucial aspect of improving the diversity in our organization is to continue transforming our systems for delivery, reliability, and accountability across the organization. This effort is a crucial step for fostering an inclusive and equitable culture. We have made a lot of progress in improving our delivery systems and will continue this improvement as an organization.

We hope that this post both makes transparent a distinct lack of diversity within our organizational leadership and provides clarity for some of our plans to improve. We are excited about the organization that we have built so far, as well as being fiercely proud of our open-source culture and development practises. We still know that we must do better, and we’re fully committed to doing so over the coming years.

We often refer to the Project Include documentation in creating an inclusive culture and set of policies. ↩︎

Ephemeral Interactive Computing for NASA Communities

Thu, 27 Jun 2024 00:00:00 +0000

We are pleased to announce that we have deployed two ephemeral hubs for NASA communities!

What did we do? #

As part of the deliverables for our NASA High Priority Open-Source Science (HPOSS) award, we deployed two new ephemeral hubs:

a public small BinderHub that offers a “reader” experience where learners can interactively view GitHub repositories that deliver light scientific content with small compute and no barriers to authentication
a big BinderHub that offers an “explorer” experience where learners can log in to access more substantial compute resources to interactively investigate large datasets and run large workflows from any GitHub repository.

These services enrich the interactive computing ecosystem for NASA communities by

improving the shareability and reproducibility of scientific information
broadening participation for historically excluded and under-resourced science communities
enabling activities such as hackathons, demonstrations or training, during workshops and conferences.

How did we do it? #

Ephemeral interactive computing services benefited from some of our previous development work in collaboration with GESIS (see our detailed blog post for more information). The research and development of this project achieved wide-reaching impact across many NASA communities we currently serve, including ScienceCore, Openscapes, US Greenhouse Gas Center, VEDA and CryoCloud; as well as networks beyond the NASA scope, such as the NSF-funded Project Pythia and HHMI-funded Spyglass projects.

What next? #

We will focus on bolstering the community- and knowledge-building needed for making the best use of these binder services in the next phase of our HPOSS work to accelerate broader participation in science. This includes opportunities such as running workshops and tutorials, as well as disseminating best practices for collaborative research. Further engineering developments will proceed in collaboration with the NASA VEDA project to set up a binder service, improve the sharing of custom environments, and refine QGIS integrations for analysing geospatial data.

Can I use this ephemeral hub service? #

The answer is yes!

For the public small BinderHub anyone can view GitHub repositories that deliver light scientific content with small compute and no barriers to authentication
For the big BinderHub you will need to be member of a NASA community. This requires a GitHub account for membership of the GitHub Team 2i2c-nasa-binder-access:big-binder-team for authorization. Please send us an email at binder-requests@2i2c.org to be added to the GitHub Team.

Acknowledgements #

NASA NSPIRES F.15 High Priority Open-Source Science Award NNH22ZDA001N-HPOSS
The NASA ScienceCore community (originally seeded by TOPS-T award 80NSSC23K0861), an anchor user community whose training delivery requirements helped shape the design of these ephemeral binder services

Neurohackademy Summer School in Neuroimaging and Data Science 2024

Fri, 21 Jun 2024 00:00:00 +0000

Neurohackademy Summer School

2i2c are pleased to support the Neurohackademy Summer School in neuroimaging and data science again!

Following the success of our collaboration in previous years (see the event page for 2023), this year’s course will be held July 29th – August 10th, 2024 and will be hosted by the University of Washington eScience Institute.

We provide an interactive computing platform for participants to get hands on experience in data pipelining, machine learning and data visualization techniques. Take a look at the following links to learn more about the neurohackathon:

Acknowledgements #

Funded by grant R25MH112480 from the US National Institute of Mental Health awarded to Ariel Rokem and Noah Benson.

The Neurohackademy Summer School is sponsored by

US Greenhouse Gas Center supports summer school at CIRA

Thu, 20 Jun 2024 00:00:00 +0000

Summer school for inverse modeling of greenhouse gases 2024

The Cooperative Institute for Research in the Atmosphere ( CIRA) is an interdisciplinary cooperation between NOAA research scientists and Colorado State University. CIRA is hosting a summer school for inverse modeling of greenhouse gases using atmospheric data assimilation techniques. The US Greenhouse Gas Center is supporting the workshop by providing 40+ attendees access to their interactive computing hub operated by 2i2c (see our blog post about their launch).

Hub administrators have built a customized software environment with container technology for use at the workshop. In doing so, this bypasses the need for participants to individually install software on their own machines and the online hub provides a consistent and reproducible interactive computing environment that is easily accessible and scalable.

Acknowledgements #

2i2c would like to credit the following individuals for their great efforts supporting this workshop:

Sanjay Bhangar ( Development seed)
Slesa Adhikari (NASA IMPACT)

Hacking the Project Pythia Cook-off with MyST Markdown

Tue, 18 Jun 2024 00:00:00 +0000

Photo courtesy of Dr Debanjana Das

What is Project Pythia? #

Project Pythia is the education working group for Pangeo, a community platform for Big Data geoscience in which 2i2c operates a cloud hub. The core aim of Project Pythia is to spearhead the creation and curation of community-driven, open-source documentation, in the form of “cookbooks”, to enable the adoption of open, scalable and reproducible workflows for geoscientists.

What did 2i2c do? #

Jenny, James and Angus from the 2i2c team participated in the annual Project Pythia Cook-off 2024, a hackathon where cookbook authors and collaborators can spend dedicated time on creating and maintaining their content using Jupyter Book and deploying their cookbooks with GitHub actions.

2i2c teamed up with the infrastructure breakout group during the hackathon, led by Katelyn FitzGerald ( UCAR) and Kevin Tyle ( University at Albany), and members of the Curvenote team also joined the group.

Day 1 #

2i2c deployed and demonstrated a dedicated BinderHub service for Project Pythia that allowed hackathon participants to “self-serve” images of their software environment, which were specified by including a list of packages in an environment.yml file placed in their GitHub cookbook repository. Participants could then pull the image from a container registry into their 2i2c hub (or indeed, any other JupyterHub server) to share and reproduce their computational environments with ease.

Day 2 #

During the first half of the day, we quickly identified a number of issues that were proving to be a maintenance headache for the Project Pythia infrastructure group:

Configuration files for each cookbook were difficult to update at scale. Project Pythia currently have a gallery of over 30 cookbooks!
Changes to Sphinx-based themes inherited from upstream were prone to breaking custom Project Pythia branding downstream.
Executable content was not able to run on the Project Pythia’s dedicated BinderHub hosted on JetStream2 (operated by NSF).
Cookbooks frequently cross-referenced materials from other cookbooks to build upon pre-existing knowledge, but this was not easy to author and the reader experience was not as smooth as it could be.

Following the announcement that Jupyter Book 2.0 will use MyST last month, Rowan (Curvenote) and Angus (2i2c) delivered a compelling demonstration of the MyST ecosystem centered around modern web-first technologies (JavaScript/TypeScript) that offers improved interactivity and accessibility.

In the second half of the day, we decided to use the hackathon to explore migrating the Pythia cookbooks from using a Sphinx-based to a MyST-based document structure and engine. Within one afternoon, the group migrated four cookbooks to use MyST MD

This moment was palpably exciting! It was evident that MyST MD supported backwards compatible content out of the box, which alleviated fears of sunk cost into existing Sphinx-based cookbooks. The migration workflow was as simple as executing the following commands

conda install mystmd

myst.

Day 3 #

We spent this day tackling support for managing a gallery of Project Pythia cookbooks at scale. See the Executable Books blog post for technical details on how we

Centralized configuration
Prototyped a gallery plugin in Python
Fixed a number of bugs related to integrated computation with Binder and JupyterLite
Embraced the referencing and reuse of content with simple markdown syntax for hover-references.

Day 4 #

Looking to the future, we spent time reflecting on our experiences and discussing the potential, transformative impact MyST MD tooling could have in the hands of the scientific community at large, including the communities served by 2i2c. Knowledge-sharing based on static figures and PDFs would fall obsolete and give way to a dynamic, web-first approach to sharing interactive narratives backed by compute from a Jupyter server.

Throughout the course of the hackathon, the rate of iterated development for both end users of the community cookbook and the developers of the open-source tooling was astounding. For example, we were able to quickly expose small bugs ( e.g. support for HTML video tags) in the MyST MD tooling, which were immediately fixed upstream and released within minutes. The feedback loop that connected the user experience with the software tooling was incredibly synergistic, with immediate impact both upstream and downstream that 2i2c hopes to continue replicating across many facets of their operations.

Beyond the Project Pythia Cook-off, the breakout group will continue conversations around strengthening their community of practice and hopefully advocating for wider adoption of MyST MD amongst the scientific community (say hello to some of our group members at SciPy 2024 in July!).

Acknowledgements #

University at Albany (NSF award 2324302): Led the funding acquisition, helped organize and facilitate the event
UCAR (NSF award 2324303): Led the planning and logistics for the event
Project Pythia for organizing this workshop.
Jupyter Book for providing development and collaboration at this workshop.
2i2c / Code for Science and Society (NSF award 2324304): Provided tailored compute services and on-site support
Curvenote: Contributed engineering cycles to MyST MD development.

Low storage alerting for the UToronto cluster

Fri, 14 Jun 2024 00:00:00 +0000

The UToronto hub landing page

2i2c has operated The University of Toronto hub since 2021 and this hub supports over 6000 educators and learners in a day! With a community of this size, file storage can quickly grow out of control and cause issues.

The 2i2c engineering team have implemented a low storage alerting system for Microsoft Azure, so that they can pre-emptively take remedial action before the filesystem is about to run out of diskspace.

Great job team 🚀

UToronto hub usage

Howard Hughes Medical Institute publishes preprint on Spyglass, a framework for reproducible and shareable neuroscience research

Tue, 21 May 2024 00:00:00 +0000

Spyglass landing page

Spyglass is a framework for reproducible and shareable neuroscience research produced by Loren Frank’s lab at the University of California, San Francisco. They recently released a preprint about their toolbox, and are using a 2i2c hub to provide accessible interactive cloud environments that demonstrate its functionality and helps researchers get started.

Acknowledgements #

Thanks to HHMI for funding and collaborating on this work.

Researchers at LEAP-Pangeo investigate overlooked sub-grid air-sea heat flux in climate models

Tue, 21 May 2024 00:00:00 +0000

Figure from the preprint showing large and small scale air-sea fluxes are separated. By Julius Busecke et al., licensed under CC BY 4.0

Julius Busecke et al. of the LEAP-Pangeo¹ hub, have recently published a preprint² that investigates “The Overlooked Sub-Grid Air-Sea Flux in Climate Models” using 2i2c infrastructure.

See Julius’ social media post for a more bite-sized outline of the work done.

Well done all! 🎉

Acknowledgements #

This research was conducted using the Pangeo ecosystem and infrastructure provided by 2i2c.

NSF Science and Technology Center (STC) Learning the Earth with Artificial intelligence and Physics (LEAP) (Award # 2019625) ↩︎
doi.org/10.31223/X5WQ47 ↩︎

A QGIS desktop in the cloud with JupyterHub

Sat, 05 Aug 2023 00:00:00 +0000

The QGreenland Researcher Workshop

JupyterHub is a versatile platform that can serve a desktop with Geospatial Information Systems (GIS) software in the cloud. This was demonstrated by the QGreenland Researcher Workshop that was hosted by the NASA CryoCloud hub. The hands-on workshop trained 25-30 researchers, from Germany, India, France, Canada, Poland and the United States, on how to work with geospatial data in an open science framework.

QGreenland Overview #

QGreenland is an open-source geospatial data package designed for QGIS, a community-owned GIS platform. It focuses on Greenland, offering researchers and educators a comprehensive toolset for FAIR (findable, accessible, interoperable and reproducible) data analysis. The package integrates a variety of datasets into a single, easy-to-use data-viewing and analysis platform, supporting both offline and online use. This makes it particularly valuable for remote fieldwork and areas with limited internet access.

Workshop Success #

The QGreenland workshop demonstrated several key benefits of using JupyterHub for cloud-based GIS:

Accessibility: Participants from across the world could access the same powerful GIS tools through a web browser, eliminating the need for complex local installations while enhancing reproducibility
Cloud block storage: Using a JupyterHub in the cloud allowed for faster data access than a traditional NFS file store by provisioning each user with an elastic block store disk, reducing load times from 5 minutes to under 3 seconds.
Cost Efficiency: Utilizing the CryoCloud JupyterHub instance managed by 2i2c drastically cut down setup costs and time, with only minimal cloud operating expenses of roughly $1/person/day.

Conclusion #

The success of the QGreenland workshop underscores the potential of integrating interactive software applications in JupyterHub. This approach not only democratizes access to advanced geospatial tools but also fosters a collaborative research environment. We look forward to supporting more workshops for QGreenland in the future!

Want to know more? Check out the companion post by QGreenland on the Jupyter Blog

Acknowledgements #

Trey Stafford (CIRES)
Matthew Fisher (CIRES)
*Fisher, M., *T. Stafford, T. Moon, and A. Thurber (2023). QGreenland (v3) [software], National Snow and Ice Data Center.
Snow, Tasha, Millstein, Joanna, Scheick, Jessica, Sauthoff, Wilson, Leong, Wei Ji, Colliander, James, Pérez, Fernando, James Munroe, Felikson, Denis, Sutterley, Tyler, & Siegfried, Matthew. (2023). CryoCloud JupyterBook (2023.01.26). Zenodo. 10.5281/zenodo.7576602

* Denotes co-equal lead authorship

2i2c partners with UC Berkeley and CloudBank to provide data science education hubs for community colleges in California

Fri, 01 Jan 2021 00:00:00 +0000

Data8. MIT licensed.

Data8 began as a large introductory data science class at UC Berkeley. It uses a Jupyter Book for all course materials, and uses JupyterHub magic links to distribute course content from the textbook.

2i2c is working with the Data 8 team to deploy JupyterHubs for community colleges in California that run the Data8 course, to make the infrastructure and content broadly accessible.

Giving people curricula, content, and infrastructure goes a long way to adoption. Communities want to remix content for their specific needs.

Acknowledgements #

This effort is funded by 2i2c together with UC Berkeley and CloudBank.

Jenny Wong

Mon, 01 Jan 0001 00:00:00 +0000

Jenny is an Open Source Applications Engineer focused on delivering the Platform and Services roadmap for 2i2c. Her previous work has included the CZI-funded Catalyst Project, an initiative aiming to provide research communities in Latin America and Africa with access to large-scale scientific infrastructure. She’s built community-based training materials for interactive cloud-native workflows. Jenny is responsible for building user-centered open source applications that ensure the alignment of Product and Services value streams to the needs of the communities we serve.

As a former research software engineer, Jenny has considerable experience supporting researchers with writing documentation, creating video tutorials, and designing and delivering training for advanced research computing. Jenny is also a qualified Software Carpentries Instructor.

Jenny’s background is in mathematics, with a special interest in geophysical fluid dynamics. She obtained a PhD in the subject with her thesis titled “A slurry model of the F-layer in the Earth’s core”.

Jenny Wong | 2i2c

Report from the Jupyter Security Working Group security tooling sprint

Acknowledgements #

Better sharing UX with nbgitpuller and contextual error handling

TL;DR #

What is nbgitpuller? #

How does nbgitpuller work? #

When it goes wrong #

Merge conflicts #

Error UX (old) #

Error UX (new) #

Learn more #

Acknowledgements #

Adding User Group Insights to Cloud Cost Dashboards with Grafana

Learn more #

Acknowledgements #

NASA Openscapes mentors run airborne data machine learning workshop with 401 participants from 68 countries

Learn more #

Acknowledgements #

Enabling transparent cloud cost monitoring with user-level dashboards

Learn more #

Acknowledgements #

Announcing `jupyterhub-groups-exporter`: monitor usage based on JupyterHub group membership with Prometheus and Grafana

Motivation #

JupyterHub and user groups #

Exporting user group memberships to Prometheus #

Visualizing user group resource usage with Grafana #

Future work #

Acknowledgements #

Announcing the Jupyter Book 2 alpha

Install the Jupyter Book 2 Alpha #

New Features in 2.0 alpha #

Rich Hover Previews #

Share & Embed Content #

Simple Instant Search #

High Quality PDFs #

Coming Soon in 2.0 beta #

Custom Styles & Scrips #

Generate Markdown from Code Cells #

Control Cell Visibility with Tags #

Jupyter Book 2 vs MyST-MD #

Acknowledgements #

Track and manage cloud costs using Grafana

Accessing the cloud cost dashboard #

Understanding the cloud cost dashboard #

Sharing cost reports #

Next steps #

Acknowledgements #

Introducing the Catalyst Project Community Partner Highlights

Highlighting the Catalyst Project Community Partners #

African Institute of Biomedical Science and Technology (AiBST) #

Malawi University of Science and Technology (MUST) #

MolerHealth #

Nelson Mandela African Institution of Science and Technology (NM-AIST) #

Nodo Nacional de Bioinformática (NNB-CCG) #

Instituto Nacional de Enfermedades Respiratorias (INER) #

Centro Interdisciplinario en Ciencia de Datos y Aprendizaje Automático (CICADA) #

About the blog posts – Acknowledgements #

Acknowledgements #

MyST Mini-Hackathon with the DeepLabCut Team

The DeepLabCut Team #

Jupyter Book and MyST #

Summary #

Acknowledgments #

NeuroHackademy Summer School Reflections

What is NeuroHackademy? #

Global and inclusive #

Collaboration with 2i2c #

Previous years #

This year #

Next year #

Acknowledgements #

References #

Keeping PACE with GPU enabled compute to detect global cloud cover using satellite data

Machine Learning with GPUs #

Managing shared memory on 2i2c hubs #

Conclusion #

References and Acknowledgments #

Collaborating with Development Seed to deliver cyberinfrastructure for NASA VEDA

Better image management and testing #

New Features in `2.0 alpha` #

Coming Soon in `2.0 beta` #