I actually live in canada. It is kind of my problem because I live with my mother.
This site is for US law.
While there are many thing that US and Canadian law have in common, there are also differences.
Were you in the US, my advice would be to:
1) Your mother should file a police report about the threats. (So far you've only mentioned threats of property damage, specifically your mother's.)
2) Start keeping a journal about your household's interactions with your brother.
3) Ditto AJ's suggestion of cameras.
4) Look into family therapy.
Not knowing the details of the situation, I can only say that yes, he can *try* to sue. Depending on the merits of his claim, he may or may not win. Vandalizing his mother's property might affect his credibility.
If there was an oral agreement to reimburse/pay him, and your mom didn't keep up her end of the agreement, then I can understand your brother's POV.