Can you tell us more about your experiment setup? What power is the transmitter set to, how far apart, what kind of antenna (or attenuators and cables), etc.? What's the squelch set to, and which tone(s)?
This "AAAA" thing you're talking about sounds a lot like the "talk off" problem described here (look for a whole section on "talk off" and related problems. These problems are in the transmitter, not in the receiver. If the transmitter is working correctly (has the right filters in the right stages doing their job well) transmitted audio can't interfere with the sub-audible tone.
There's other useful information at the site I linked above, like avoiding tones that are close to harmonics of your AC power mains, and using tones between 127.3 Hz and 173.8 Hz.
Since you say the transmitter was opengd77 also we could look at if the AT1846S is really configured correctly for transmit (especially that the right filters are enabled). It lets you configure it incorrectly if you think you have a reason to, but I think it's already correct. You could be running into limitations of the (transmitter) hardware.
I'm also basically suspicious of how the whole squelch system in OpenGD77 is set up. It's doing a lot of things in firmware that the AT1846S is capable of doing for itself, and which I suspect it's probably better at than we are.
I also don't quite grasp why we rely so heavily on the traditional squelch rather than solely or mostly on the tone and code squelches when they are configured. I get that noise can trigger the decoder but that's why we wait until we see the decoder hold a while before we open and why we hang a while after the decoder loses the signal (unless we have STE, which is not implemented yet in the firmware but the AT1846S supports).
I've not been doing a lot of OpenGD77 hacking lately, so someone else may want to experiment with these things.