To prevent VLC from resizing the window every time you open a new video:

VLC -> Preferences -> Show All -> Interface/Main interface/macosx

Uncheck the option Resize interface to the native video size

vlc

vlc