If the court order said the father has primary custody then he still does until the court rules otherwise. Just because he let the child "go stay with mom" for awhile does not mean he has given her custody. He can go get his child as long as he does not disobey any court rulings of visitation...