Well I have noticed worse with phones. People having their phones out during a conversation and starting to reply or look at something on their phone while "listening" to me talk. This was at school, however during the summer when I work people with smartwatches I would notice would typically excuse themselves if they noticed an important phone call or message they had to deal with. They would always finish talking though before they even checked if it was important though, unless it was buzzing like a phone call.

But like you said, this is also my personal experience and my school and work environment may be much different than yours, and even have a different kind of cultural approach.