If he is their legal father (born while you were married), then he has as much right to see them as you do. You cannot prohibit it, since you both have legal custody.
Has anyone filed for divorce?
Contact an attorney to find out about filing for emergency or temporary custody and support.