create packaged models from checkpoint

[AdrianKs]

Allows to create a packaged model, which only contains the model, entity/relation ids and the config.
It is not possible to resume training on a packaged model but it can be evaluated.

example command:
kge package checkpoint_best.pt

normal checkpoints also store entity/relation ids now

[rgemulla]

How are pacakged models used? Maybe we should rename KgeModel.load_from_checkpoint to KgeModel.load_from and support checkpoints and packages (including creation of required "virtual" datasets) there.

[rgemulla]

TrainingJob.load should offload some functionality to KgeModel.load_from. And I guess KgeModel.load_from should accept both filenames and loaded files (to facilitate this).

[AdrianKs]

TrainingJob.load should offload some functionality to KgeModel.load_from. And I guess KgeModel.load_from should accept both filenames and loaded files (to facilitate this).

This could be done, but I am not sure if we should do it.
KgeModel.load_from creates a new model based on the config file provided in the checkpoint.
At the time we call TrainingJob.load we already have a model created which we just update with the parameters in the checkpoint.

If we want to move this to the load_from function we would also need to provide the new config file and there we unnecessary create a new model.
We would only offload the functionality of the model loading to load_from

This creates an overhead with no big advantages.

[rgemulla]

I see. We still need load_from to work with packages (that's a key point of having packages). And we should try to minimize code duplication. Ideas welcome!

[AdrianKs]

This does not help much with the code duplication but with the designflaw having the evaluation job use the trainjob resume.
lets move most of the content from trainjob.resume to the new method job.resume.
In job.resume we check which checkpoint we need to load and load it.

trainjob.resume then looks something like this:
def resume(self, checkpoint_file) \ checkpoint = super().resume(checkpoint_file) \ self.load(checkpoint)
evaljob.resume then just looks the same
but the load functions can be different

this does not help much with the load and load_from distinction but in general would be a lot cleaner, even if there is maybe even more duplication.
but the train job types are more independent.

[AdrianKs]

actually also the self.load(checkpoint) call could be done in job.resume
then we only have a job specific load function.

[rgemulla]

Sounds good. But note that resuming from a checkpoint generally may need to use the config from the checkpoint, not the one from the folder (which may not even exist). When cleaning up the API, this should be considered. We'd need sth. like TrainingJob.load_from(checkpoint) and KgeModel.load_from(checkpoint) as static methods. One advantage is that the training job then does not need to create a model first, it simply calls KgeModel.load_from. Also, the methods should accept both a filename or a loaded checkpoint in their "checkpoint" argument. This way, we'd only need to load the checkpoint once.

[AdrianKs]

in that case the function resume() could maybe completly be replaced by the static method load_from, since currently we create a new model and then call resume and afterwards run() anyways. This could be repaced by job = load_from(checkpoint) and afterwards job.run().

[rgemulla]

Sounds like a good idea to me.

[AdrianKs]

I built a first version to replace the method resume with load_from. This actually resulted in more changes than expected. Please have a look, if this is worth the changes.

[AdrianKs]

With these changes we could also make it possible to evaluate checkpoints without the config, I think

[rgemulla]

I think it's worth it. Perhaps some of the more generic checkpoint-handling code can be put into a seaparate package (kge.util.io?).

In general, I think we should have KgeModel.load_from_checkpoint(cp), Job.load_from_checkpoint(cp), Dataset.load_from_checkpoint(cp), and Config.load_from_checkpoint(cp). If we also had the reverse methods (save_to_checkpoint) everywhere, we could remove code clutter and code duplication.

[AdrianKs]

I tried to address most of your points.

[rgemulla]

I think valid is a sensible default choice.

[AdrianKs]

It is now possible to evaluate without a config by calling EntityRankingJob.create_from(checkpoint).

If we later want to change the console api to enable evaluation without a config, we would have to find a way to still support additional commandline options without overwriting the checkpoint config with all the default values

[AdrianKs]

the loading of the dataset should now work as expected. In KgeModel.create_from we just always set preload_false. In that way we can create a model without the dataset. If the datasets are later on needed it will throw an IOError

with that everything should be adressed

[rgemulla]

Looks very good. I added a few final points, mostly ,minor.

[rgemulla]

map_indexes seems to be incorrect here. Do you mean map_indexes? Also, why not just store meta["key"]?

[AdrianKs]

I switched to map_indexes. If we just store meta["key"], we can not make sure, that we actually loaded this data. Could be that we didn't even read the entitiy_ids file, when calling this method.

[rgemulla]

What's the current state here? Can you rebase this off the current master?

[AdrianKs]

separated loading of checkpoint from create_from and rebased.
Should be done now.

[rgemulla]

I made some revisions. Let me know your thoughts. If you are fine, we can merge this PR.

[AdrianKs]

Thanks, the revisions look good. I'd say we are ready to merge

[rgemulla]

Thanks!