now of course if i take some psychical action towards him i can get in trouble right ?
Yes. We call that the crime of battery in California.
or can i break into his house late and night and stand over his bed and then leave and not get in trouble right ?
Who said he was not going to get in trouble? I said he would not be arrested right away because the law says he cannot. If the DA presses charges he could be charged, arrested, tried, and convicted for up to a year in jail just for the trespassing in the occupied dwelling offense alone!
If you break in to HIS house, and he brains you with a bat or shoots you, you'll be injured or dead AND potentially face criminal charges. This would be a very dumb thing to do.
If your mom, sister, or sister's friend want to make a point, they can all get restraining orders against him. That way, if he ever does it again, he can go to jail. And for the ex-girlfriend, hers would be a domestic violence order which can allow the police to arrest based upon probable cause even if he was gone when they got there.
also if they pursue a restraining order what r the chances of them aactually getting it cause im pretty sure they dont just hand them out u have to have resonable proof that they need a restraining order right ? like physical proof
They do not need physical proof, only sufficient proof to convince a judge. With both their statements, a police report and investigation, damage to the doggy door, and - with luck - his own admission to being there, a restrainin order should be rather easy. If he denies being present and there is no proof of the entry aside from the broken doggy door and the statements of two people that might have a beef against him, then a judge might not be compelled to issue the order. but, they will not know if they do not try. And, chances are, that if he broke into your house and into your sister's room just to get a glimpse of his ex-girlfriend, he has done some weird stuff before that can add to the affidavit.
If any of the involved girls are minors, that will help, too.
But,
YOU do not have standing to do anything here. From what you wrote, the house belongs to mom so she is a victim and your sister and her friend (the ex-girlfriend) are witnesses.