Pangeo

Last updated Jun 22, 2022 Edit Source

• Pangeo - A community platform for Big Data geoscience
• https://github.com/pangeo-data/education-material
• https://github.com/pangeo-data/awesome-open-climate-science
• https://github.com/pangeo-data/ml-workflow-examples
• https://github.com/pangeo-data/
• https://discourse.pangeo.io/
• Pangeo Forge
  • Pangeo Forge is an open source platform for data Extraction, Transformation, and Loading (ETL). The goal of Pangeo Forge is to make it easy to extract data from traditional data repositories and deposit in cloud object storage in analysis-ready, cloud-optimized (ARCO) format
  • https://pangeo-forge.org/catalog

• #CODE Pangeo
• #CODE Rechunker - Disk-to-disk chunk transformation for chunked arrays
  • https://rechunker.readthedocs.io/en/latest/
• #CODE Climpred - Verification of weather and climate forecasts
  • https://climpred.readthedocs.io/en/stable/index.html
  • Metrics for forecast verification
• #CODE xESMF: Universal Regridder for Geospatial Data
  • https://pangeo-xesmf.readthedocs.io/en/latest/
  • Comparison of 5 regridding algorithms

• Pangeo for DL

• Streaming with Zarr

• #PAPER The PANGEO Big Data Ecosystem and its use at CNES (Eynard-Bontemps 2019)

  • https://medium.com/pangeo/why-and-how-we-use-pangeo-at-cnes-74553c7fb19b

• #PAPER The Pangeo Ecosystem: Interactive Computing Tools for the Geosciences: Benchmarking on HPC (Odaka 2019)

  • https://archimer.ifremer.fr/doc/00597/70946/
  • #CODE https://github.com/pangeo-data/benchmarking

See AI/DS and DataEng/Dask

1
2
3
4

    # For the dask dashboard
	from dask.distributed import Client
	client = Client()
	display(client)

See AI/DS and DataEng/Xarray

• https://medium.com/pangeo/thoughts-on-the-state-of-xarray-within-the-broader-scientific-python-ecosystem-5cee3c59cd2b
• Slow performance of open_mfdataset
  • https://xarray.pydata.org/en/stable/io.html#reading-multi-file-datasets

See AI/DS and DataEng/Jupyter