read_csv does not work for species name with a comma

[BangShiuh]

Problem description

read_csv does not properly handle a comma in a species name.
Ex. CH(1,3).

gas = ct.Solution("gri30_mod.yaml")
states = ct.SolutionArray(gas)
gas.TPX = 300, ct.one_atm, "O2:1.0"
states.append(gas.state)
states.write_csv("output.csv")
states2 = ct.SolutionArray(gas)
states2.read_csv("output.csv")

Line #2 (got "x" columns instead of "x+1")
x is the number of variables.

System information

• Cantera version: 3.0.0

Attachments

Additional context

[ischoegl]

@BangShiuh ... I think that csv being a 'comma-separated-list' by definition/default, this behavior is expected. So I do not believe that this is improper handling of output, but a problematic (i.e. difficult-to-parse) choice of species name.

I'd recommend using a different output format (although even YAML may face similar issues; HDF may not), or replace the comma in the species name with a different character.

[bryanwweber]

I'm surprised this isn't handled by the write_csv method. You can quote values with commas in CSV files so that this doesn't happen. Doesn't write_csv use NumPy or pandas underneath?

[ischoegl]

write_csv uses the csv package, while read_csv uses numpy.genfromtxt (with delimiter set to ','). I believe there are other internal writers in use beyond what is defined in composite.py, so there isn't a uniform approach to CSV. I.e. I don't think that there can be any guarantees for a species name that contains a comma at the moment.

[bryanwweber]

I think this is worth fixing... This is a common enough convention for species naming to indicate a particular shape of the molecule, so I don't see "don't use those species" as a long term solution. As I said this is a known/solved problem if you have to embed the delimiter in the value for any *sv type file, so it may just take a little more configuration to resolve on the write side.

[ischoegl]

I think this is worth fixing

I don’t have an issue with this, but would like to see solutions that are not API-centric.

[corykinney]

I think this is worth fixing

I don’t have an issue with this, but would like to see solutions that are not API-centric.

Could you clarify what you mean when you say API-centric?

I think this could be easily solved by switching from np.genfromtxt to pd.read_csv, as the latter supports commas enclosed in quotes, which is the widely accepted approach (see point 6 in the definition of the CSV format) to formatting fields with commas in them.

[bryanwweber]

Thanks for the link @cory-kinney! I think we probably don't want to introduce a hard dependency on pandas to this code. This link suggests a fix using the csv module which seems like it would work, although we'd probably want to adapt it for our specific use: https://stackoverflow.com/a/41930090

I'm not sure what Ingmar meant about API specific. I suspect he means that ideally the fix would be able to work for all the code that needs to read csv input, not just this one place in the Python interface.

[ischoegl]

I'm not sure what Ingmar meant about API specific. I suspect he means that ideally the fix would be able to work for all the code that needs to read csv input, not just this one place in the Python interface.

#1385 introduces a new C++ SolutionArray that will work for any API. At the moment, it only covers Sim1D for HDF and YAML formats, but follow-up work will establish a link to the Python loaders, where support for CSV will be relatively straight-forward.

PS: likewise thanks for the link … I haven’t found a simple CSV library for C++ (and I’m not sure that it’s worth introducing a dependency for a relatively simple task), so this document is a nice reference

[bryanwweber]

I haven’t found a simple CSV library for C++

Looks like it should be fairly simple with boost: https://bravenewmethod.com/2016/09/17/quick-and-robust-c-csv-reader-with-boost/

[corykinney]

Of course! I found the link on Stack Overflow, naturally.

Looks like it should be fairly simple with boost: https://bravenewmethod.com/2016/09/17/quick-and-robust-c-csv-reader-with-boost/

This looks like it would cover the main edge cases that we would encounter. No need to reinvent the wheel.

[ischoegl]

This looks like it would cover the main edge cases that we would encounter. No need to reinvent the wheel.

Yup. PR’s are always welcome 😜

[speth]

There should be no need to use Boost for this, by the way -- the C++ standard library now includes <regex> (which we use elsewhere) and has I believe essentially the same API as the Boost version. See https://en.cppreference.com/w/cpp/regex. Anything available in C++14 or older is fair game as far as Cantera is concerned.

[ischoegl]

Now that #1426 is merged, all the infrastructure is in place to handle this in C++ as part of SolutionArray. I created a new feature request in Cantera/enhancements#163, as I currently don't have the bandwidth to tackle this myself.