While using ssh your connection may become idle or unresponsive in any case instead of waiting you can simple terminate the connection by sending a escape sequence:

~.

SSH escape sequences

sequence description
~. terminate connection (and any multiplexed sessions)
~B send a BREAK to the remote system
~C open a command line
~R request rekey
~V/v decrease/increase verbosity (LogLevel)
~^Z suspend ssh
~# list forwarded connections
~& background ssh (when waiting for connections to terminate)
~? this message
~~ send the escape character by typing it twice