While using ssh your connection may become idle or unresponsive in any case instead of waiting you can simple terminate the connection by sending a escape sequence:

~.

🔗SSH escape sequences

sequencedescription
~.terminate connection (and any multiplexed sessions)
~Bsend a BREAK to the remote system
~Copen a command line
~Rrequest rekey
~V/vdecrease/increase verbosity (LogLevel)
~^Zsuspend ssh
~#list forwarded connections
~&background ssh (when waiting for connections to terminate)
~?this message
~~send the escape character by typing it twice