What to do if a client is asking for a refund because they don’t want to sign a permit for work legally requiring one?

He has no grounds to sue you: the law is very plain that you *cannot* contract or agree to break the law, so if a permit is required, your refusal to violate the law by not applying for one is NOT valid grounds to terminate the contract or escape the "no refund" clause. You can't be required to do that which is illegal. It is not theft because theft requires a criminal mens rea or intent, which you do not have. Furthermore, since the permit is so cheap and you are paying, there was no loss or harm to him, which means that the failure to advise him was not material (important or critical) and so would not support either breach of contract or fraud claims. In short, based on what you write, he does not have any valid claims against you for refusing to violate the law or not disclosing an unimportant factor, and he is bound to the no-refunds contract he signed.
You could reciprocate if he does sue: countersue him for "malicious use of process" or "abuse of process" (differents states use different terms for this cause of action), for abusing the legal process to try to extort money from you and seek sanctions against his attorney for knowingly bringing a frivolous lawsuit (lawyers can't bring lawsuits that they know are not supported by the facts and/or law).
Or, you may decide that even though you are in the right, that it makes more economic and emotional sense to try to settle for returning his entire deposit and moving on. If you can settle, get the settlement in writing before paying.