When wearing scarf with jacket, where does it sit?

I've been wearing my scarf[1] inside my jacket collar for years, but I suddenly wondered if outside would hold more warmth?

[1] yes, the same scarf[2] that's in my profile pic for many many years!

[2] by @fbz from knityak