[Question]: Why is expand_context set to false when training (TransformerEmbeddings)?

[kobiche]

Question

I was debugging the behaviour of the model during training, I landed on this peculiar behaviour.
If I understand it correctly, the expand_context is set to false if we are training the model. Why? Why is the context expanded only during inference?

[alanakbik]

Hello @kobiche the context is always expanded during inference. During training, we apply a "context dropout" where the context is only expanded if the random number is above the dropout value. The idea is to make the model robust to work without context as well (for instance, if someone wants to predict for only a single sentence.)

[kobiche]

Thanks for the clarification, I think I misinterpreted the if-condition.