Because swingers do not wear T-shirts which says we like to have sex with other couples. Your boring neighbours could be wild swingers or the elderly couple, sitting on the terrace all day.
Sometimes this is only between the women, but most of the time, the men like some interaction with the woman of the other couple as well.
Swingers can also do so-called full group sex swap, where they go all the way, this means sexual intercourse with penetration.
Although in the old days this was a bit easier, because most swingers were shaving their intimate parts completely.
During daytime the difference between a hairy intimate part and a bold one, can be spotted from miles away, so to speak.
Swingers are usually more open, they make eye contact, they are flirting with each other or they display some sexual behaviour.