You start on your harmony note, it sounds right, and four bars later you are singing the tune. Everyone who learns harmony goes through this, and it is not a sign of a bad ear. It is a sign of a good one: your ear is locking onto the strongest line it can hear, which is the melody, and your voice is following your ear.

The fix is not to listen less. It is to give your own line enough weight in your head that the melody cannot take it over. That comes from two things: knowing your line cold before the tune starts, and practising holding a note against something that moves.

The drill: one note against a moving tune

Hold a single note, the tonic, while a tune moves over it. Nothing to find, nothing to follow; the only job is to not move. The tune will pull, especially where it lands on your note and then leaves. Stay. Play this, solo the lower line to hear how little it does, then sing it while the tune runs.

Then a note against a leaping tune

The same idea with a tune that jumps around. Leaps are where your ear is most likely to let go of your note and grab the tune's. If you can hold this one, you can hold a third.

Then the real thing

Once the drone holds, go back to a third above and notice where it slips: it is nearly always the same spot, usually a leap, or a long note where the tune is sitting on something more interesting than yours. Sing your line alone through that spot ten times. Then with the tune.

Blend's drone lessons are this drill, graded: it tells you the bar where your note moved and which way. The third-above lessons after them show every note you sang against the one you were meant to, so “I drifted” becomes “I drifted in bar three, on the leap.”