Skip to content

ODC SC Meeting 2020 01 30

Luigi Di Fraia edited this page Jan 29, 2020 · 16 revisions

Attendees

  • Luigi di Fraia (Catapult SA)
  • Federica Moscato (Catapult SA)
  • Robert Woodcock (CSIRO)
  • Syed Rizvi (AMA)
  • Paul Haesler (CSIRO)
  • Kirill Kouzoubov (GA)
  • Alex Leith (GA)
  • David Gavin (GA)
  • Randall Sunne (USGS)
  • Tony Butzer (USGS)
  • Andrew Cherry (AMA)
  • John Rattz (AMA) (For Syed)
  • Chris Morgan (FrontierSI)
  • Peter Wang (Data61)
  • George Dyke (Symbios)

Apologies

Minutes

See Google Drive document.

Agenda

  • Welcome
  • Previous Actions
  • State of the Cube
    • CSIRO

      • Continuing work on CSIRO data-pipelines (live ingester):
        • Generic prepare scripts.
          • Loads all product definitions into memory.
          • Auto matches a source dataset to a product definition and picks the one with the highest score.
          • User mostly needs to provide metadata extraction for the dataset.
        • Testing real time operation on a single node before deploying to EASI.
          • Subscription based and launches a series of daily orders.
          • Decouples ordering from download/processing.
          • Historical catch up, then switch to real time.
    • Catapult

      • Continuing work on ARD processing workflows:
        • Landsat 5-8
        • Sentinel-2
        • Sentinel-1 (including complications around the anti-meridian over Fiji)
      • Continuing work on CommonSensing Notebooks.
      • Working on approaches to leverage our Dask cluster in the above mentioned Notebooks.
      • Working on a Cube REST API for product generation.
    • GA

      • Alex Leith attended the Radiant Earth ML Workshop and had a discussion with some folks from Pangeo there. Alex would like to develop a dynamic STAC API off the ODC database, so that we can integrate in a light way with the Pangeo stack.
      • Work is underway preparing data for the ODC Hackathon over Ghana. It will be Sentinel-1, Sentinel-2 and Landsat collection 1.
      • We're working with Element 84 on converting Sentinel-2 data over Africa to COGs with STAC in us-west-2
      • We continue to work with datacube-alchemist, which is a scene-to-scene WOfS and FC processor (but can be used for more algorithms). Current focus is doing Sentinel-2 data in Australia on AWS.
      • And we're transitioning super computers, migrating processes to the new one called Gadi.
      • Test infrastructure is now fully containerised (./check-code.sh --with-docker will run all the tests on dev machine with only dev machine requirement being -- "docker is installed and user has permissions to run it")
      • Using Github Actions instead of TravisCI for continuous intergation testing, typical CI tests time is down to around 5 minutes (15-20 was common previously)
      • Improvements for dockerized workflows: datacube can be configured via single environment variable now, no need to write out config files via entry point or other orchestration means.
    • USGS

    • AMA

      • Open source release of updated UI, and several new notebooks.
      • Working to download LS7 & LS8 2018-2019 data for the 5 ARDC countries.
      • Working to deploy a light-weight ODC Sandbox with demo notebooks.
      • Working on creating a connection between Google Earth Engine and Open Data Cube.
      • Providing support to ARDC training with AWS infra. and up-to-date UN SDG notebooks.
      • Working to move ARDC from ingested data to indexed data.
      • Working with other countries wanting to use ODC, e.g., Peru, El Salvador, etc.
      • Other projects - California Cube, Virginia Cube, etc.
    • FrontierSI

  • Enhancement Proposals
    • New Proposals
    • Committed Proposals
  • ODC V1.8 and V2.0
  • Hack-a-thon
    • 24-28 February 2020
  • Other Business
  • Next Meeting and Close
    • TBA

Actions

See the action tracking project.

Outstanding Actions

New Actions

  • Example
Clone this wiki locally