Large references need to be defined as such in the same plugin that adds the reference in order to work correctly.
If only overwriting ESM plugins define a reference to be large, it might either trigger large reference bugs and/or the full model may not loading/unloading correctly, even inside the active exterior cells.
See large reference for more explanations.