If you don't have evidence or witnesses(who are willing to speak up), then yes it comes down to your word against his and either of you could lie or exaggerate. In fact, if your stories don't match then someone has to be lying, so how can anyone else know.