I think I’ve settled on the latter. Disagreement is maybe best communicated by the absence of an upvote? And downvotes work best when they signal something that is just off base, and while not reportable, is not appreciated at a broad cultural level.
Somewhat vague but I think of it as “this doesn’t belong here.” It seems to be the most fitting - something could not belong because it’s irrelevant, or because it’s rude, or because nobody wants to see it. All up to interpretation, I suppose, but better that than a hard rule than I either don’t feel good about sometimes or that prevents me from downvoting things that probably should be downvoted but don’t explicitly break that rule.