This is a good question. I'm wondering whether this man really wants to "take you to court" and would think this is all just words to scare you. Is he paying child support? Does he even have a stable income? Where would the child stay?
My understanding is that he may have an equal custody right but he does not have the right to remain in the United States and could be deported. If he tries to use the system, it is possible that the authorities will be notified and he'll have another problem. It is possible that a court could award him custody if it is truly in "the best interests of the child" but unless I hear different, I would think that this is probably not likely to happen. If he actually does follow through, you would probably want to hire a family law attorney and you may wish to also have an immigration lawyer involved in the discussions as well.