Add Hugging Face integration

[NielsRogge]

Hi @dbolya,

Thanks for this nice work. I wrote a quick PoC to showcase that you can easily have integration so that you can automatically load the various Hiera models using from_pretrained (and push them using push_to_hub), track download numbers for your models (similar to models in the Transformers library), and have nice model cards on a per-model basis. It leverages the PyTorchModelHubMixin class which allows to inherits these methods.

Usage is as follows:

from cotracker.models.core.cotracker.cotracker import CoTracker2

model = CoTracker2.from_pretrained("nielsr/co-tracker-hf")

The corresponding model is here for now: https://huggingface.co/nielsr/co-tracker-hf. We could move all checkpoints to separate repos on the Meta organization if you're interested. Ideally, each checkpoint has its own model repository on the 🤗 hub, where a config.json and safetensors weights are hosted.

To improve discoverability, we could add tags like "object-tracking" to the model cards, so that people can find these models by either typing in the search bar/filtering on hf.co/models.

Would you be interested in this integration?

Kind regards,

Niels

[dbolya]

Hi @NielsRogge,

Thanks for the PR! This sounds like a good idea and definitely something we'd be interested in.

As for the implementation, I had a couple questions. Currently we have this repo set up as a PyPI package, and I would like to ensure that functionality doesn't break. I noticed in the code, you changed relative import of hiera_utils to an absolute one. Is that something that was breaking the PyTorchModelHubMixin integration? As stands, this change currently breaks the package functionality.

Also, the change requires us to make hiera-transformer depend on huggingface-hub. I wonder if it's better to only import huggingface-hub if the user actually uses that functionality. Do you know how other packages typically handle this?

[NielsRogge]

Thanks for your reply :)

Regarding changing the relative import, the only reason I was doing that was because I got:

Traceback (most recent call last):
  File "/Users/nielsrogge/Documents/python_projecten/hiera/hiera/hub_mixin.py", line 3, in <module>
    from hiera import HieraForImageClassification
  File "/Users/nielsrogge/Documents/python_projecten/hiera/hiera/hiera.py", line 31, in <module>
    from .hiera_utils import pretrained_model, conv_nd, do_pool, do_masked_conv, Unroll, Reroll
ImportError: attempted relative import with no known parent package

when attempting to run the hub_mixin.py script. I've added it back :)

Regarding making huggingface_hub a soft dependency, that's definitely possible. In the Transformers library, we handle this by checking if the library is required, and only return an import error if the library cannot be found in the environment of the user. I've pushed something similar here.

Would you be up for trying this PyTorchModelHubMixin to push the various Hiera checkpoints to the hub (perhaps as part of the Meta organization)?

[dbolya]

Thanks for the update! This looks good to me. I'll incorporate this into v0.1.3 (and add you as a contributor to that update).

In the mean time, yes it would be preferable to upload the models to the Huggingface hub under the Meta organization. However, I'm not currently a member of the facebookresearch org, so one of my coauthors will likely have to do it (e.g., @chayryali). What would that entail?

[dbolya]

Created the PR for this (#26). Also confirmed that it works (e.g., dbolya/hiera-tiny-224).

Also, another question: for this repo, we have it set up so that you have the main model (.e.g., hiera-tiny-224) and various checkpoints (e.g., mae_in1k_ft_in1k) with one checkpoint being the default. I noticed you could do the same with huggingface hub branches (e.g., dbolya/hiera-tiny-224@mae_in1k_ft_in1k), but is there a way to set one branch as default? I.e., so we can explicitly control which checkpoint is used when you omit the branch name.

[NielsRogge]

Awesome, thanks a lot :)

Regarding pushing to the Meta organization, yes your coauthor could do that assuming he/she has write access, or you could be added to the organization as well assuming you still work at Meta.

Regarding the branches, the from_pretrained and push_to_hub methods support the revision argument, which allow you to specify a commit hash or branch name, e.g.:

from hiera import Hiera

#Load from hub
model = Hiera.from_pretrained("dbolya/hiera-tiny-224", revision="b56f9b4aa964ddfc4690f3b901fe61f00bdccc38")

Based on the commit history of your repo. Since each repo on the hub is a git-based repo, you could create various branches for a single repo, and specify the branch name. However, usually we just create one separate repository per model checkpoint (to have separate model cards, download numbers, etc.)

[NielsRogge]

Will close this one in favor of #26