I think brandishing is a bit more flagrant than open carry in a holster. Open carry would not make me nervous, but brandishing might. I think I have seen one person open carrying in 8 years and hundreds of hikes. No idea how many concealed carriers. Never once brandishing though - I'm certain of that.
I thought we we primarily talking about open carry. If someone carries concealed, how would I know to avoid them?