Are you getting called for interviews and then not getting the job? Or are you not getting called for interviews?
If the first, then it's POSSIBLE that this is affecting your ability to get a job. If that's so, it's a shame but as long as what they are saying is the truth, it's legal.
If the second, there is NO chance whatsoever that this is in any way a factor, since employers do not call for references on applicants until they have ALREADY been interviewed, often more than once, and made the short list of the top two or three candidates. In fact, in some instances, employers only do reference checks on their first choice candidate until or unless s/he turns down the job, when they go to the next choice, and so on. But they NEVER call previous employers before deciding who to interview. No one has the time and resources to do that, especially these days.