In classical Athens, crimes fell into one of two categories. There were private crimes such as theft, assault, and murder. Charges for these crimes could only be brought by the victim, although, in the case of murder, obviously this would have to include the victim's family.
Then there were public crimes, which were considered to harm the state as a whole. Any citizen could prosecute one of these crimes. In this category were included treason, extortion of state funds, and... adultery!
Someone who committed adultery in Athens at this time certainly might have tried to hide the fact, by, say, denying they had done so. But if accused, one thing they never would have done would be to tell the accuser "It's none of your business."
That answer would have made as little sense to a contemporary Athenian as it would today to tell the police, about the shooting victim lying on your living room rug, "It's none of your business."