pims | James Colliander

Syzygy, Callysto and 2i2c

Tue, 25 Feb 2020 23:00:00 +0000

PIMS and Initiative Planet A

Wed, 16 Oct 2019 05:26:35 +0000

Canadians land on Jupyter

Fri, 24 Aug 2018 11:55:38 +0000

National Scale Interactive Computing

Wed, 22 Aug 2018 19:32:16 +0000

(Invited post on Jupyter blog: https://blog.jupyter.org/national-scale-interactive-computing-2c104455e062)

This is an invited post from Jim Colliander, Professor of Mathematics at UBC and Director of the Pacific Institute for the Mathematical Sciences.¹

In 2017, the Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched Syzygy, a cloud-hosted interactive computing platform that delivers JupyterHub deployments for universities across Canada.

Syzygy has been used by over 16,000 students at 20 universities. The main results of the Syzygy experiment so far are:

Demand for interactive computing is ubiquitous² and growing strongly at universities.
The Jupyter ecosystem is an effective way to deliver interactive computing.
A scalable, sustainable, and cost-effective interactive computing service for universities is needed as soon as possible.

Demand for interactive computing

Both research and teaching at universities are adapting to major societal changes driven by explosions in data and computational tools. New educational programs that prepare students to think computationally are emerging, while research strategies are changing in ways that are more open, reproducible, collaborative, and interdisciplinary. These transformations are inextricably linked and are accelerating demand for interactive computing. The Syzygy experiment has shown that using Jupyter in educational programs drives interest in using Jupyter for research (and vice versa). For example, students in mathematics, statistics, and computer science collaborated with a researcher from St. Paul’s Hospital in Vancouver using Syzygy to identify new pathways to prevent death from sepsis. Research communities typically need access to deeper computational resources and often span multiple universities, but the common thread is the need to expand access to interactive computing.

A map of JupyterHub deployments deployed by Syzygy.

Technical milestone achieved

The Syzygy project has demonstrated that it’s possible to deploy tools for interactive computation at a national scale rapidly and efficiently using an entirely open source technology stack. Students, faculty and staff across Canada use Syzygy to access Jupyter through their browsers with their university single-sign-on credentials. The JupyterHubs range from a “standard” configuration to bespoke environments with specially curated tools and data integrations. This richness is possible because of the architecture of the Syzygy and Jupyter projects and the flexibility of the underlying cloud resources. As a case-study, Syzygy demonstrates that the Jupyter community has achieved a significant technical milestone: interactive computing can be delivered at national scale using cloud technologies.

Service level requirements

The validation that interactive computing can be technically delivered at national scale prompts universities to ask a variety of questions. Can interactive computing service be delivered robustly? How will users be supported? What are the uptime expectations? What is the data security policy? How is privacy protected? Can the robustness of the service be clarified in a service level agreement? Syzygy, as an experimental service offered to universities at no charge and without a service level agreement, does not properly address these questions. To advance on their education-research-service mission and address growing demand, universities need a reliable interactive computing service with a service level agreement.

What’s next?

How should universities address their needs for interactive computing over the next five years? Right now, universities are following two primary approaches:

🙏 Ad hoc: faculty figure out how to meet their own needs for interactive computing; IT staff deploys JupyterHub on local or commercial cloud servers; this approach gives universities control over their deployments and hardware, though requires time and expertise that many may not have.
🎩 Use a cloud provider’s service: Google Colab, Amazon Sagemaker, Microsoft Azure Notebooks, IBM Watson Studio; this approach allows universities to quickly launch interactive computing services, with a loss of flexibility and some risks by becoming reliant upon a particular vendor’s closed-source and proprietary software.

These approaches are not sustainable over the long term. If universities all deploy their own JupyterHub services, many will need technical expertise they do not currently have and will involve a significant duplication of effort. If universities rely on hosted cloud notebook services, the reliance on proprietary technology will impair their ability to switch between different cloud vendors, change hardware, customize software, etc. Vendor lock-in will limit the ability of universities to respond to changes in price for the service. Universities will lose agility in responding to changes in faculty, staff, and student computing needs.

There is a third option that addresses the issues with these two approaches and generates other benefits for universities:

🤔 Form an interactive computing consortium: universities collaborate to build an interactive computing service provider aligned with their missions to better serve their students, facilitate research, and avoid risks associated with vendor lock-in.

To retain control over their interactive computing stacks, avoid dependence⁴ on cloud providers, and accelerate the emergence of new programs, universities should work together to deploy interactive computing environments in a vendor-agnostic manner. This might take the form of a consortium — an organization dedicated to serving the needs of universities through customized shared infrastructure for interactive computing. The consortium would also ensure that universities will continue to play a leadership role in the development of the interactive computing tools used for education and research.

The Syzygy experiment confirmed that growing demand for interactive computation within universities can be supplied with the available technologies advanced by the Jupyter open source community. In the coming year, we aim to build upon the success of the Syzygy experiment and seed an initial node of a consortium in Canada with the intention of fostering a global network of people invested in advanced interactive computing. If you are interesting in partnering, please get in touch! See this Jupyter Community Forum post to continue the discussion.

The author gratefully acknowledges feedback on this piece from Ian Allison, Lindsey Heagy, Chris Holdgraf, Fernando Perez, and Lindsay Sill.
Interactive computing needs have been identified in agriculture, applied mathematics, astronomy, chemistry, climate science, computer science, data science, digital humanities, ecology, economics, engineering, genomics, geoscience, health sciences, K-12 education, neuroscience, political science, physics, pure mathematics, statistics, and sociology.
Relying on commercial cloud vendors to provide the interactive computing service for universities risks recreating the problems associated with scientific publishing that emerged with the internet.

Canadians Land on Jupyter

Tue, 11 Jul 2017 17:00:00 +0000

The proliferation of mobile devices, social networks and sensor networks, the massive data streams they generate, and increasing computational power generate research challenges and provoke widespread interest in mathematical sciences expertise and insight. Democratizing access to this expertise and insight catalyzes meaningful change. Higher education institutions are launching interdisciplinary programs in digital humanities, data science, scientific computation, mathematical modeling, bioinformatics and epigentics to address these challenges. To achieve success, these programs require easy access to state-of-the-art computing environments to support research, teaching and training activities.

The Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched a cloud-hosted scientific computing and data science platform for Canada. The service, syzygy.ca, delivers Jupyter to faculty, staff and students at Canada’s universities using single-sign-on (SSO) via their university user account. By eliminating the requirement to install customized software on personal computers, syzygy.ca makes it easier for research teams to collaborate using the right tools for their investigations. The platform delivers an interactive coding environment for literate programming in Python 2, Python 3, R (and sometimes Julia, Octav, Sage and other languages).

The data8 program at Berkeley, the open source quantitative economics course, and computational fluid dynamics course are inspirational examples showcasing the potential for Jupyter. PIMS is leveraging syzygy.ca and other tools to develop expertise in scientific computing, data science, machine intelligence, optimization, etc.

Jupyter service is available today at several universites (UBC, SFU, UofT, Waterloo, Queen’s, Victoria, Saskatchewan, Manitoba, Calgary, Lethbridge). A few hubs have been deployed that can be accessed using other identity authentication providers (Google OAuth; GitHub; GitHub Enterprise). Colleges, universities, and other prospective partners can request Jupyter service via syzygy.ca by clicking here.

PIMS developed some support resources: e-book introduction; Discourse forum.

The platform has been used for seminars on Python and Git, machine learning with SciKit Learn, neural networks and deep learning, undergraduate courses on mathematical computing, computer science, and statistics, and graduate courses on mathematical modeling for industry, seismic inverse problems, and computational finance.

Academy-industry partnerships are forming to investigate data science challenges arising in business through a workshop built atop syzygy.ca.

The syzygy.ca platform democratizes access to digital research infrastructure. PIMS and our partners advance Canada’s research capacity by connecting human talent to curated tools from the mathematical sciences, diverse data sources, excellent documentation and training programs.

Budget 2017, Naylor’s review, and the Mathematical Sciences in Canada

Fri, 21 Apr 2017 19:29:59 +0000

James Colliander, Director of the Pacific Institute for the Mathematical Sciences (PIMS)
Nassif Ghoussoub, Director of the Banff International Research Station (BIRS)
Ian Hambleton, Director of the Fields Institute for Research in Mathematical Sciences (Fields)
Luc Vinet, Directeur du Centre de Recherches Mathématiques (CRM)

The direct funding of research initiatives on artificial intelligence (AI) and quantum computing via Budget 2017, and the release of the report of Canada’s Fundamental Science Review present an opportunity to reflect on the role of mathematical sciences within Canada’s scientific heritage and future, but also on our country’s ways of funding research.

The importance of the mathematical sciences (mathematics, statistics and computer science) is deepening in almost all areas of knowledge. Mathematical sciences provide a conceptual infrastructure underpinning advances in biology, engineering, humanities, medicine, social sciences and beyond. Progress in our understanding in all these fields depends upon advanced research and high-level training in the mathematical sciences.

Canadian Mathematician John Charles Fields, the creator of the Fields medal (often dubbed the Nobel prize for mathematics), also played a key role in the founding of the National Research Council in 1916. Today, Canada is served by a collaborative network of mathematical sciences research institutes: the Centre de Recherches Mathématiques (CRM) in Quebec, the Fields Institute for Research in Mathematical Sciences in Ontario, and the Pacific Institute for the Mathematical Sciences (PIMS) in Western Canada.

The institutes amplify Canada’s capacity for discovery and invention through partnerships that intertwine our nation’s universities with academic and industrial researchers from across the globe. Together, the institutes created Mitacs in 1999, which, under the leadership of Arvind Gupta, became a cornerstone of the government’s effort to link our graduate students (in all disciplines) with industry. This accomplishment was amply recognized in Budget 2017. In 2003, they collaborated with Berkeley’s Mathematical Sciences Research Institute (MSRI) to found the Banff International Research Station (BIRS), a unique North-American research infrastructure on Canadian soil, that provides an environment for creative, synergetic, intense and prolonged interactions between mathematical scientists and investigators in other areas of research.

In 2002, the three institutes committed to provide long-term funding to the Atlantic Association for Research in the Mathematical Sciences (AARMS), a network that plays an important role in the mathematical sciences research activities of the Atlantic region.

In 2003, PIMS collaborated with Berkeley’s Mathematical Sciences Research Institute (MSRI) to found the Banff International Research Station (BIRS), a unique North-American research infrastructure on Canadian soil, that provides an environment for creative, synergetic, intense and prolonged interactions between mathematical scientists and investigators in other areas of research.

The need for leadership to advance the mathematical, computational and statistical understanding of information, the development of data science, and the advent of machine learning, prompted the institutes in 2012 to use their own resources to invest in the creation of CANSSI, the Canadian Statistical Sciences Institute. That NSERC did not have the resources to do so at that time sheds some light on the community’s reaction to how Budget 2017 continues to shut out the Tri-Council.

AI rests on mathematical sciences. Indeed, some of this field’s prominent leaders pursued their foundational research within BIRS, CRM, Fields and PIMS. AI needs further advances in the mathematical sciences to thrive. We celebrate strong support of AI research but the disjointed approach used by government for its substantial investment in the area of deep learning totally missed the opportunity to include and exploit the national resource that BIRS, CRM, Fields, PIMS and CANSSI represent.

This example is but one of many examples that illustrate the importance of some of the excellent recommendations of the report of the Science Review Panel. Indeed, Canada developed world leading expertise and operations in science policy during the era of Pierre Elliot Trudeau. The work of the Senate Special Committee on Science Policy, chaired by Senator Maurice Lamontagne, identified principles to guide Canada’s future governments. The consultation of the scientific community overseen by the panel chaired by David Naylor aligns with these best practices. Direct investments disbursed through political channels, instead of through the Tri-Council’s scientific peer review process, undermine the transparency of research funding programs, and can miss opportunities such as the one we described. Funding allocations to support research should follow consistent and rigorous evaluation processes incorporating independent scientific peer review.

I am Joining PIMS and UBC Math

Mon, 28 Apr 2014 19:10:03 +0000

I’m excited to be joining the Pacific Institute of Mathematical Sciences as Deputy Director and the Department of Mathematics at the University of British Columbia.

Here is a link to the announcement: http://www.pims.math.ca/news/james-colliander-appointed-deputy-director-pims

(I have moved my blog and web presence over here: http://colliand.com/)