Automatically pick best shuffle implementation for a query

[andygrove]

What is the problem the feature request solves?

Users currently have to choose between using Comet native shuffle or Comet JVM shuffle. If they choose native shuffle and it is not supported then we fall back to Spark. If they choose JVM shuffle and it is not supported then we fall back to Spark.

It would be a better user experience if we automatically used native shuflfle when possible, then fall back to JVM shuffle when possible, and finally fall back to Spark if neither native or JVM shuffle is supported.

Describe the potential solution

No response

Additional context

No response

[vaibhawvipul]

I would like to work on this issue.