Absolutely not. The adjuster for the other driver usually has various sources of information about the accident before he or she calls you. (Police report, his insured’s account, witness statements and private investigators.) Thus, the main reason they want to talk to you is to see if you will say something that will hurt your case.
Of course, if you plan to settle your case without an attorney, then you will have to talk to the adjuster—he or she will be the one who negotiates the settlement.