Why can I still hear input signal when I'm not recording or when my recording software is not open?

This is because the interface's direct monitoring is enabled. This is the signal that goes direct from the interface's mic, line or inst input to the output without going through the computer so that you can monitor while recording without any latency (delay). If you wish to turn this off you should follow the directions for your interface below.

Interfaces that use Saffire or Scarlett MixControl

Select Routing Preset > DAW tracking

This routes DAW 1 and DAW 2 (outputs 1 and 2 of your software) to all pairs of outputs and disables 

DAW Tracking

Scarlett 2i2 and iTrack solo

Switch the direct monitor switch to "off" on the front of the hardware.

Scarlett 2i4

Set the direct monitoring knob all the way to the "Playback" side on the front of the hardware.

Forte

Mute or turn down Inputs 1 and 2 in Forte Control.

Forte Control inputs