Jupyter | James Colliander

Building the Open Source Science Stack

Wed, 22 Feb 2023 00:00:00 +0000

2i2c + Berkeley

Thu, 21 Oct 2021 09:00:19 +0000

Open Toolchain

Fri, 27 Aug 2021 10:00:00 +0000

Syzygy, Callysto and 2i2c

Tue, 25 Feb 2020 23:00:00 +0000

Communities of practice for Jupyter deployments on shared infrastructure

Thu, 07 Nov 2019 14:00:27 +0000

NSERC Open Data Exploration

Sat, 24 Nov 2018 00:00:00 +0000

NSERC Open Data Exploration

The 2018 federal budget of Canada 🇨🇦 announced significant new investments in research. The Canada Research Coordinating Committee is defining a work plan to promote “international and risky research; support for Canada’s next generation of researchers; equity, diversity and inclusion; and indigenous research and capacity building”.

This site explores the NSERC Awards Data with the goal of helping Canada to achieve the best outcomes for research and training of future researchers. Open analysis of research investments also helps advance the goals outlined in the Mandate Letter to the Minister of Science.

Canada’s Open Government Portal includes NSERC Awards Data from 1995 through 2016. The awards data (in .csv format) were copied to an Amazon Web Services S3 bucket and analyzed using computing resources from PIMS on cloud infrastructure from Compute Canada and Cybera.

You are invited to collaborate on the analysis of investments made by NSERC. Access to the data and source code are shared in a public repository on GitHub.

TODO Set up similar analyses for SSHRC

TODO Set up similar analysis for CIHR

Overview

Interactive Notebook: Exploring NSERC

NSERC Budget

Year	Total	Not Discovery	Discovery
1995	451159747	267113329	184046418
1996	434116337	247533171	186583166
1997	417146970	227810152	189336818
1998	477141539	266731883	210409656
1999	526133116	299956302	226176814
2000	537934451	294508093	243426358
2001	554698568	316537927	238160641
2002	615320903	357333281	257987622
2003	696997644	423803197	273194447
2004	765643760	488699482	276944278
2005	820640167	519567082	301073085
2006	854567999	545233090	309334909
2007	968713281	655321429	313391852
2008	980889930	664642572	316247358
2009	1003483757	680426976	323056781
2010	1026366682	702101715	324264967
2011	1035205511	705872398	329333113
2012	1021894152	685991741	335902411
2013	1018139165	681486392	336652773
2014	1038149414	698064881	340084533
2015	1068044894	727279990	340764904
2016	1142066427	791496311	350570116

Accumulated Program Investments [1995 - 2016] Top 25

ProgramNameEN	Reported Dollars
Discovery Grants Program - Individual	5949789135
Canada Research Chairs	1590512911
Collaborative Research and Development Grants	948122955
Strategic Projects - Group	908959760
Networks of Centres of Excellence	762559105
Research Tools and Instruments - Category 1 (<$150,000)	565583275
Postgraduate Scholarships	435601321
Industrial Research Chairs	398835692
Research Networks	382614308
Alexander Graham Bell Canada Graduate Scholarships - Doctoral	352888065
Postgraduate Scholarships - Doctoral	340062867
Postdoctoral Fellowships	283101969
Subatomic Physics Envelope - Project	267988202
University Undergraduate Student Research Awards	258863102
Major Resources Support Program - Infrastructure	201128769
Centres of Excellence for Commercialization and Research - Group	186214812
Engage Grants Program	185938402
Alexander Graham Bell Canada Graduate Scholarships - Master’s	175254486
Cooperative Activities	163228194
College and Community Innovation Program	161161262
Major Facilities Access Grants	159523772
Collaborative Research and Training Experience	147907095
Canada Excellence Research Chairs	137314666
Discovery Grants Program - Accelerator Supplements	121369119
Postgraduate Scholarships - Master’s	114880771

Accumulated Principal Investigator Investments [1995-2016] Top 25

Name	Reported Dollars
Hormes , Josef	99836800
Gupta , Arvind	97587570
McWalter , Ian	75503588
Fortier , Louis	72542772
Frise , Peter	63874923
Barge , Brian	50400000
Thomlinson , William	46610800
Fedosejevs , Robert	40720798
McDonald , Arthur	36781300
Orr , Robert	32043881
Smith , Donald	31227906
Mufti , Aftab	28761712
Wallace , Douglas	28433833
Servos , Mark	28130452
Rogers , Harry	27590000
Salama , Andre	26296562
Burton , Howard	25000000
Yada , Rickey	24982617
Clowes , Ronald	24072844
Pelton , Robert	23985763
Svensson , Carl	22782441
Lamb , Robert	21349800
Chrisman , Nicholas	21286250
Larter , Stephen	20024564
Bourbonnais , Normand	17753965

TODO NSERC’s Open Data does not report gender of PIs. Find ways to measure equity, diversity, inclusion metrics over past investments.
TODO How can we connect the reported dollar investments to measurements of research output?

Discovery

Interactive Notebook: Evaluation Groups Analysis

Evaluation Groups

Award Amounts

TODO Why do 1501 and 1502 receive so much more investment than other EGs?

Number of Awards

TODO Why are there so many more awards in 1501 and 1502 than in other EGs?

Bin Frequency

Example: 1504 (Chemistry); FiscalYear 2013

TODO Find a way to infer bin levels based on awards data amounts

Engage

Interactive Notebook: Engage Grants

TODO Find a way to connect Engage Grant investments to economic impact metrics.

Canadians land on Jupyter

Fri, 24 Aug 2018 11:55:38 +0000

National Scale Interactive Computing

Wed, 22 Aug 2018 19:32:16 +0000

(Invited post on Jupyter blog: https://blog.jupyter.org/national-scale-interactive-computing-2c104455e062)

This is an invited post from Jim Colliander, Professor of Mathematics at UBC and Director of the Pacific Institute for the Mathematical Sciences.¹

In 2017, the Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched Syzygy, a cloud-hosted interactive computing platform that delivers JupyterHub deployments for universities across Canada.

Syzygy has been used by over 16,000 students at 20 universities. The main results of the Syzygy experiment so far are:

Demand for interactive computing is ubiquitous² and growing strongly at universities.
The Jupyter ecosystem is an effective way to deliver interactive computing.
A scalable, sustainable, and cost-effective interactive computing service for universities is needed as soon as possible.

Demand for interactive computing

Both research and teaching at universities are adapting to major societal changes driven by explosions in data and computational tools. New educational programs that prepare students to think computationally are emerging, while research strategies are changing in ways that are more open, reproducible, collaborative, and interdisciplinary. These transformations are inextricably linked and are accelerating demand for interactive computing. The Syzygy experiment has shown that using Jupyter in educational programs drives interest in using Jupyter for research (and vice versa). For example, students in mathematics, statistics, and computer science collaborated with a researcher from St. Paul’s Hospital in Vancouver using Syzygy to identify new pathways to prevent death from sepsis. Research communities typically need access to deeper computational resources and often span multiple universities, but the common thread is the need to expand access to interactive computing.

A map of JupyterHub deployments deployed by Syzygy.

Technical milestone achieved

The Syzygy project has demonstrated that it’s possible to deploy tools for interactive computation at a national scale rapidly and efficiently using an entirely open source technology stack. Students, faculty and staff across Canada use Syzygy to access Jupyter through their browsers with their university single-sign-on credentials. The JupyterHubs range from a “standard” configuration to bespoke environments with specially curated tools and data integrations. This richness is possible because of the architecture of the Syzygy and Jupyter projects and the flexibility of the underlying cloud resources. As a case-study, Syzygy demonstrates that the Jupyter community has achieved a significant technical milestone: interactive computing can be delivered at national scale using cloud technologies.

Service level requirements

The validation that interactive computing can be technically delivered at national scale prompts universities to ask a variety of questions. Can interactive computing service be delivered robustly? How will users be supported? What are the uptime expectations? What is the data security policy? How is privacy protected? Can the robustness of the service be clarified in a service level agreement? Syzygy, as an experimental service offered to universities at no charge and without a service level agreement, does not properly address these questions. To advance on their education-research-service mission and address growing demand, universities need a reliable interactive computing service with a service level agreement.

What’s next?

How should universities address their needs for interactive computing over the next five years? Right now, universities are following two primary approaches:

🙏 Ad hoc: faculty figure out how to meet their own needs for interactive computing; IT staff deploys JupyterHub on local or commercial cloud servers; this approach gives universities control over their deployments and hardware, though requires time and expertise that many may not have.
🎩 Use a cloud provider’s service: Google Colab, Amazon Sagemaker, Microsoft Azure Notebooks, IBM Watson Studio; this approach allows universities to quickly launch interactive computing services, with a loss of flexibility and some risks by becoming reliant upon a particular vendor’s closed-source and proprietary software.

These approaches are not sustainable over the long term. If universities all deploy their own JupyterHub services, many will need technical expertise they do not currently have and will involve a significant duplication of effort. If universities rely on hosted cloud notebook services, the reliance on proprietary technology will impair their ability to switch between different cloud vendors, change hardware, customize software, etc. Vendor lock-in will limit the ability of universities to respond to changes in price for the service. Universities will lose agility in responding to changes in faculty, staff, and student computing needs.

There is a third option that addresses the issues with these two approaches and generates other benefits for universities:

🤔 Form an interactive computing consortium: universities collaborate to build an interactive computing service provider aligned with their missions to better serve their students, facilitate research, and avoid risks associated with vendor lock-in.

To retain control over their interactive computing stacks, avoid dependence⁴ on cloud providers, and accelerate the emergence of new programs, universities should work together to deploy interactive computing environments in a vendor-agnostic manner. This might take the form of a consortium — an organization dedicated to serving the needs of universities through customized shared infrastructure for interactive computing. The consortium would also ensure that universities will continue to play a leadership role in the development of the interactive computing tools used for education and research.

The Syzygy experiment confirmed that growing demand for interactive computation within universities can be supplied with the available technologies advanced by the Jupyter open source community. In the coming year, we aim to build upon the success of the Syzygy experiment and seed an initial node of a consortium in Canada with the intention of fostering a global network of people invested in advanced interactive computing. If you are interesting in partnering, please get in touch! See this Jupyter Community Forum post to continue the discussion.

The author gratefully acknowledges feedback on this piece from Ian Allison, Lindsey Heagy, Chris Holdgraf, Fernando Perez, and Lindsay Sill.
Interactive computing needs have been identified in agriculture, applied mathematics, astronomy, chemistry, climate science, computer science, data science, digital humanities, ecology, economics, engineering, genomics, geoscience, health sciences, K-12 education, neuroscience, political science, physics, pure mathematics, statistics, and sociology.
Relying on commercial cloud vendors to provide the interactive computing service for universities risks recreating the problems associated with scientific publishing that emerged with the internet.

Canadians Land on Jupyter

Tue, 11 Jul 2017 17:00:00 +0000

The proliferation of mobile devices, social networks and sensor networks, the massive data streams they generate, and increasing computational power generate research challenges and provoke widespread interest in mathematical sciences expertise and insight. Democratizing access to this expertise and insight catalyzes meaningful change. Higher education institutions are launching interdisciplinary programs in digital humanities, data science, scientific computation, mathematical modeling, bioinformatics and epigentics to address these challenges. To achieve success, these programs require easy access to state-of-the-art computing environments to support research, teaching and training activities.

The Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched a cloud-hosted scientific computing and data science platform for Canada. The service, syzygy.ca, delivers Jupyter to faculty, staff and students at Canada’s universities using single-sign-on (SSO) via their university user account. By eliminating the requirement to install customized software on personal computers, syzygy.ca makes it easier for research teams to collaborate using the right tools for their investigations. The platform delivers an interactive coding environment for literate programming in Python 2, Python 3, R (and sometimes Julia, Octav, Sage and other languages).

The data8 program at Berkeley, the open source quantitative economics course, and computational fluid dynamics course are inspirational examples showcasing the potential for Jupyter. PIMS is leveraging syzygy.ca and other tools to develop expertise in scientific computing, data science, machine intelligence, optimization, etc.

Jupyter service is available today at several universites (UBC, SFU, UofT, Waterloo, Queen’s, Victoria, Saskatchewan, Manitoba, Calgary, Lethbridge). A few hubs have been deployed that can be accessed using other identity authentication providers (Google OAuth; GitHub; GitHub Enterprise). Colleges, universities, and other prospective partners can request Jupyter service via syzygy.ca by clicking here.

PIMS developed some support resources: e-book introduction; Discourse forum.

The platform has been used for seminars on Python and Git, machine learning with SciKit Learn, neural networks and deep learning, undergraduate courses on mathematical computing, computer science, and statistics, and graduate courses on mathematical modeling for industry, seismic inverse problems, and computational finance.

Academy-industry partnerships are forming to investigate data science challenges arising in business through a workshop built atop syzygy.ca.

The syzygy.ca platform democratizes access to digital research infrastructure. PIMS and our partners advance Canada’s research capacity by connecting human talent to curated tools from the mathematical sciences, diverse data sources, excellent documentation and training programs.

Calculating MRR over variable term subscriptions using Pandas

Thu, 10 Mar 2016 20:20:39 +0000