Is it legal to withhold money from an invoice if that person owes you money?

Unless, there is some type of agreement allowing for this sort of deduction, then no it is not legal. One thing has nothing to do with the other.  You are owed on your invoice for work performed/materials provided by you, etc.  However, this does not mean that you do not owe this other person any money. If you two can't straighten this out, then the other broker can take you to court to enforce his legal rights (if any).  If he prevails in a lawsuit, then a judgement will be granted and the broker can then proceed to collect accordingly. If however, he has no legitimate claim to this money, then you would have to file in court and sue for its return, but the broker could then file a counter-claim. Bottom line, trying to reach some agreement between the two of you may well provide for the best solution here. It will save money on legal costs, not to mention your time.

Note:  If you do work out an agreement, be sure to put it in writing and get it  signed.

If there was an agreement giving one the right to deduct money owed by the second from payments to be made to the second, then this could be done. Without an agreement, it could not be done unilaterally--though if you sued to enforce the contract and collect what is owed you, he could then interpose his counterclaim in court and if he won, he'd then have a legally enforceable offset. Therefore, IF this was a legitimate debt, while he technically could not simply take it the way he did, if you end up in litigation, he would likely ultimately end up being awarded it. Therefore, if it is legitimate, it's in your interest to work something out with him, memorandizing that the debt has no been paid and is clear, etc. If it's not a legitimate debt, that's a different matter--he would have no entitlement to the money and you could sue for its return. If "in between," so that it's disputed--i.e. he legimitately feels its owed and has at least some evidence, just as you legitimately feel it's not owed and have some evidence--the two of you would probably do best by settling or compromising somewhere in the middle, rather than incurring the cost of litigation.