Being poor and not potty training a special needs child by the time a child without those challenges might have reached that milestone does not a case for neglect make. He might not be the best father, but parents do have a lot of latitude in how they raise their children. That includes hairstyling, bathing, and who they spend time with. If CPS isn't finding cause to remove the child, you are not going to be able to get custody without his agreement. You can enforce the order that exists, but as you have been told in the past, you need to go to court to do that. If you are unwilling to take that step, you are out of options.