How did your mother decide your children should live with her? Did you go to court and assign her as a guardian? If you didn't go through the court, nothing that you two wrote up between each other is going to be valid.
How did the police just take the kids? Was there some kind of court order...