This is not a line issues as an older modem put in place works fine.

I am not very familiar with the concepts behind modem handshaking and data transfer, hence my questions:

1. What causes a modem to connect but immediately disconnect?
2. If this is an issue with carrier quality, are there any settings in a modem that can be tuned? Should we be waiting for a longer time period before determining we have no carrier?
3. Are there any troubleshooting procedures that can be carried out to determine if the modem itself is defective? Two modems have been tried at this site, to no avail.

Any recommended reading on Modem concepts?

If modems cannot establish carrier there will be disconnect. This can happen if both modems can't agree on protocols to use.

Normally, V.90 modem is used to dial out.
If you are RECEIVING calls, you need to disable V.90 (and use V.34 or lower protocol) because a V.90 client modem cannot connect with V.90 if it is the answering modem.
www.modemsite.com/56k/ms.asp
has commands to disable V.90.