No, it doesn't seem anyone took advantage of you. You simply failed to assert your rights and seem to be deflecting all responsibility to the mother. The things you are saying are unfair have been there for you to address and enforce all along.

You signed a piece of paper you thought had some legal meaning and chose to take no further action, but you could have at any time.