Project Pythia at AGU

[kmpaul]

Project Pythia will have a presence at AGU 2021! Please provide your input here on how best we can present Project Pythia at AGU.

NS013: Open-source software for near-surface geophysics and its applications

@kmpaul has been invited to present about Project Pythia at the AGU session NS013 ("Open-source software for near-surface geophysics and its applications"). Kevin is currently working on an abstract for that session, but the thinking is that it will be essentially a "Project Pythia Overview" talk with modifications to target it to the solid-earth geoscientists in the audience. Once a draft for that session abstract is developed, it will be shared here.

IN030: Open-Source Software, Notebooks and FAIR Software

In our most recent Project Pythia meeting, we also discussed having @kmpaul (since he will be attending AGU) submit an abstract to session IN030 ("Open-Source Software, Notebooks and FAIR Software"). This would be a Project Pythia overview presentation. Here is a draft of a possible abstract, very similar to the abstract @brian-rose gave for recent Pangeo showcase:

Project Pythia is an NSF-funded EarthCube project to create a "go-to" resource for geoscientists to learn the ins and outs of the myriad technologies that comprise the scientific Python ecosystem. Partnered closely with the Pangeo community, Project Pythia aims to make it easier for geoscientists at all career levels to adopt and contribute back to the Pangeo scientific Python software stack. In this talk, I will give an overview of the current state of the project -- including the Project Pythia Portal and the Project Pythia Foundations Book, plans for the future, and how you can get involved.

What do people think?

[kmpaul]

I received confirmation from Lindsey Heagy that the IN030 session would be a great fit for a Pythia overview talk.

[brian-rose]

Great! I say go ahead with the submission to IN030. The draft abstract looks good to me.

[kmpaul]

Who should be listed as a co-author on this submission? @ProjectPythia/core @ProjectPythia/education @ProjectPythia/infrastructure

[kmpaul]

Incidentally, I've received feedback from Lindsey Heagy about the NS013 session, and it sounds like they would like a little bit of an "introduction to Pangeo", too. So, I will tailor an abstract for that session to give a bit of an introduction to Pangeo and Pythia and how the two fit together.

[kmpaul]

For NS013: "Open-source software for near-surface geophysics and its applications"

Based on feedback from Lindsey Heagy, here's what I now know about the invited talk for the near-surface geophysics community:

• "What are some of the tipping points you have seen in the Pangeo / Pythia communities that have driven the adoption of cloud computing for research and education? Any comments you have on when HPC is well suited for a problem and when cloud is suited for a problem would also be valuable."
  • Possible Answers:
    1. When the data gets "too heavy to move," doing your analysis "next to the data" is key. (Cite extended "distributed computer memory hierarchy" chart) Global climate and high-resolution meteorology data are representative of this problem and require "proximate analysis." Pangeo technology is scalable.
    2. Cloud computing offers a robust solution for both sharing data as well as services, something that local computing (including private clusters and HPC) just can't provide.
    3. Cloud computing (and increasingly HPC as well) is based on containerized technology and workflows, which makes it incredibly easy to deploy and tear down computing instances, such as for training, tutorials, classes, etc. (This also makes it possible for cloud computing environments to be elastic, and scale out only as far as you need when you need it.)
  • Other answers?
• Lindsey also said that the near-surface geophysics community uses Jupyter a lot, but they don't use Xarray, Dask, and Zarr. So, an overview of these technologies and what they are good for would be good.
  • I think the answer to this is straightforward: Xarray is "Pandas for N-dimensional arrays." Dask provides distributed scalability "under the hood" in a way that is flexible for different kinds of analyses. And Zarr provides a storage format that is more cloud friendly than HDF5.
• Lastly, Lindsey said that the number of Python packages in the near-surface geophysics community is growing rapidly, and that lessons on how Pangeo has worked to support and grow development would be appreciated.
  • I think the answer to this is to describe how Pangeo advocates and supports the community move to *open development." And so a slide on the "What/Why/How" of open development" would be good.

What do people think about these answers?

[dcamron]

Maybe less relevant for the abstract in particular, but I really like the initial question and am curious to hear about people's experiences around "tipping points" they've encountered. Are there any large-scale problems, projects, or needs where any of us had to bring in a new technology or capability to be able to move forward? Or where a leap in technology was required to even begin a new undertaking? I think a collection of experiences like this could be valuable to some communities or people looking to advocate for these sorts of shifts in technology.

[kmpaul]

Considering the above, here's a...

Draft Abstract for NS013

Title: Pangeo and Project Pythia: Helping Geoscientists Navigate the Scientific Python Ecosystem

Abstract:

Pangeo is a community of academic scientists and industry software engineers partnered to "foster collaboration around the open source scientific Python ecosystem for ocean, atmosphere, land, and climate science." While Pangeo is probably best known for its scalable software stack built on Jupyter, Xarray, and Dask, the Pangeo community's (arguably) greatest contribution is its support for and advocacy of open development practices and reproducible science. To aid in Pangeo's efforts to educate scientists and advance open science, Project Pythia was formed to created a "go-to" resource to help geoscientists navigate and learn the complex scientific Python ecosystem and to help train scientists in the best practices necessary to foster open science and open development. One of Project Pythia's most recent efforts is to build a Foundations Book, covering the basic technology and knowledge necessary to wade into the complex scientific Python ecosystem. In this presentation, I will introduce the audience to Pangeo, its software stack, and the current state of Project Pythia. I will attempt to cover how Pangeo can be useful to other communities, as well as how you can get involved.

Funding for Pangeo and Project Pythia both come from NSF EarthCube awards 1740633, 1740648, 2026863, and 2026899.

[kmpaul]

Who wants to be a co-author on this? Please just give a 👍 reaction to this post if you do. @ProjectPythia/core @ProjectPythia/education @ProjectPythia/infrastructure

[kmpaul]

@ProjectPythia/core @ProjectPythia/education @ProjectPythia/infrastructure

Just a reminder to everyone that I need to submit these abstracts by tomorrow (Aug 4, 2021), end of the business day (mountain time). I am assuming that the co-authors on these presentations will be:

• Anderson Banihirwe
• Drew Camron
• John Clyne
• Orhan Eroglu
• Max Grover
• Julia Kent
• Ryan May
• Brian Rose
• Kevin Tyle

However, I don't know if anyone wants someone else listed, too. Please let me know before tomorrow!

[kmpaul]

I have submitted both the IN030 and the NS013 abstracts.

[kmpaul]

Here is a link to the content for the AGU "Open Source Software, Notebooks, and FAIR Software" session poster.

https://docs.google.com/document/d/1sXX0EZxtrqFQqcPeEywF5RgOsXJWJix4-XrDNQjbAWo/edit?usp=sharing

Let me know what you think. I will add this content into one of AGU's iPoster templates to create an "interactive poster" for the conference session.

[clyne]

Love it! Great work!

[brian-rose]

Looks terrific!

Just one niggling comment: can you list my (and @ktyle's) affiliation as "University at Albany (SUNY)". And, if possible at this stage, I think it would be appropriate to add @r-ford to the author list.

[kmpaul]

Thanks. Sorry I keep getting the institution name wrong.

[kmpaul]

Here is a link to the AGU "Open-Source Software for Near-Surface Geophysics and Its Applications" session poster on Pangeo and Pythia:

https://docs.google.com/document/d/1j9CShwdsAaNi-iAzO3wDoSWFbCAhSjFaBsRmwTuEUM8/edit?usp=sharing

Let me know what you think. (Same deal with the AGU iPosters.)

[ktyle]

@kmpaul regarding the Pangeo/Pythia iPoster, can you add the Pythia Twitter link since you have it for Pangeo?

[ktyle]

For the Project Pythia iPoster:

Introduction panel:

1. "parent's" --> "parents'"
2. "have scares" --> "have scared"

The Platform panel:

1. "we our goal" --> "our goal"

Get Involved panel:

1. As with the Pangeo/Pythia iPoster, add the Pythia Twitter link.

Great job on both iPosters ... love the changing graphics!

[kmpaul]

Thank you!

[kmpaul]

@ktyle: All changes have been made.

[brian-rose]

@kmpaul both posters look great and read well! I really have nothing to add. Great use of the i-poster format (my device keeps autocorrecting this to imposter, ha ha)..

[kmpaul]

Thanks for the feedback, everyone! The posters have now been "published." I can make changes until Dec 31 (I think), but the posters are published in preparation for the conference.

[clyne]

Hi All,

I've been asked to give a ~4 minute talk on Project Pythia at the EarthCube Town Hall. I'd appreciate any feedback on my slides. I'll have to upload them late tomorrow. Sorry for the short fuse!

EarthCube Town Hall at AGU
Friday, 17 December 2021
11:15 - 12:15 CST
Convention Center - Room 352

[brian-rose]

Looks great for a pithy talk! I had the same nits and @ktyle about the bullets on slide 2.

[ktyle]

@clyne On slide 2, is there a reason why the final bullet, Using Cloud resources is in bold? Also I see a speaker icon at the bottom right of that slide. Finally, if only because I'm pretty focused on Git / GitHub these days, the "H" in GitHub is typically capitalized. Other than these minor nitpicks it all looks good!

[clyne]

Thanks for catching these. I've addressed them.