I agree: it does not sound like there is a restraining order against you.
However, you are also not the legal parent, and if the legal parents have made this agreement, then for the sake of smooth co-parenting your legal children with your ex, you shouldn't knowingly make trouble.
Of course...