Added data conversion scripts.

[frobnitzem]

These scripts outline the general format for working will all csv file types.

I will add to this PR as I test these general scripts on multiple data sources.

[frobnitzem]

yaml_to_config.py needs checking to ensure that it's selecting the right variables for compatibility with other datasets used to train the foundational model, and also to check the indexing it uses for graph output properties.

Specifically, it now contains:

      "input_node_feature_names": [
        "atomC",
        "atomF",
        "atomH",
        "atomN",
        "atomO",
        "atomS",
        "atomHg",
        "atomCl",
        "atomicnumber",
        "IsAromatic",
        "HSP",
        "HSP2",
        "HSP3",
        "Hprop"
      ],

But likely we can't use all these. How should yaml_to_config be modified?

[frobnitzem]

Several steps need to be done before this is ready to merge:

• implement config.json reader according to data.md
  • check that these files are correctly generated by the present PR (utils/yaml_to_config.py)
  • update the output head configuration (see below)
  • test data loading from AdiosDataset with new feature selectors
• add classification loss head type to HydraGNN
  use sigmoid as activation function at the last layer (only needed if we end with an activation, since torch's cross-entropy converts (-inf, inf) to logits.
  use binary cross entropy as loss function for training, validation, and testing
• update data loading (from AdiosDataset) so that metadata is used to target which features go into x and y (need to select only some columns when going from data file to in-memory format)

[frobnitzem]

Documenting the new format

1. List all relevant functions encountered during config loading and parsing.
2. Check steps that process the config object against the previous and new formats we are using – what needs to be changed?
3. Develop hand-written examples of the new config.json files we expect. Compare old and new formats side-by-side and explain the changes.
4. Write a pydantic BaseModel and validate our new config.json files.
   • This can now be used to fix yaml_to_config.py and create config.json files for all our downstream tasks.

Implementing the new format

1. Write down a list of all old variable names that existed in config.json before, but which are now going to be obsolete.
2. Write and test the main functions inside HydraGNN that will load the new config format.

• involves updating the data loading step to make use of feature indices now present
• involves updating the loss computation step

3. Search through HydraGNN's code and remove all these variable references. This will require testing against existing HydraGNN examples to ensure that functionality is not lost or broken.
4. Make a list of all examples that use the old config.json format. (to document future work that needs to be done without getting side-tracked doing it immediately)
5. Incorporate our pydantic.BaseModel for loading config.json into HydraGNN.

After completing the above (new issue/PR)

1. Run downstream task training.
2. Run a foundation model task training (using a smaller dataset like qm7x).
   • This will require rebuilding the dataset to include our metadata about features.
3. Run a hybrid training
   • this will be easier using the new config and data formats BUT will require re-configuring output heads and creating x,y,z values (maybe all zero-s) for input molecule.