If you want to do this the easy and least expensive way, I suggest you work with him on this.
He's right.
He has paternal rights, whether he exercises them or not!
He can't stop you, but he can get you in trouble if you take the child out of state.
Both of you must obey the existing custody order.
If the order describes where you are to keep the child (they all have such language), that is where the child must be kept.
You, of course, are free to go anywhere you choose.
The child must be kept within the confines of the county, within the state that has jurisdiction, or some other set geographic boundary.
You can go to court and request the order to be changed.
But, if dad objects, the child won't be allowed to go.
To do as you desire, you'll have to offer to pay ALL expenses associated with the child being sent back to visit dad. That assumes, of course, that dad gets on board with this change.
In the end, it is up to the court.
That means you'll need the assistance of an attorney.
The American Bar Assiciation tries to offer pro bono legal assistance to service member families in your position.
Google their website.
Warning: Don't take the child without going through the process described above. If you do, you could face serious criminal charges. You could also end up losing custody altogether.