You could get away with only speaking when spoken to. After all, dinner conversation is not a requirement if you let everyone else do all the talking for you.

[ Which is more or a less approval for Zangetsu simply not talking. Pretending to be outright mute, well, that could be easily disproven, but still. ]