The station where Brief Encounter was filmed is now a pleasant heritage centre and a place to see occasional steam trains heading to the Lake District or the Dales. More often though you will just get the blur of an electric high speed train passing through without slowing.
After a pleasant coffee there, the thing is to head up the hill a few hundred yards to the amazing secondhand bookshop, which is one of the very best that I know. This is what makes Carnforth worth a visit to my mind.
I agree with your notion of avoiding possible conflicts, yet we see marriages such as between Mary Matlin (right wing extremist) and James Carville (left wing extremist) which turn out pretty successful by all accounts.