What to do if I turned in a ounce bar of what I thought was gold to a coin shop and they payed me but later it was found that it was not really gold?

If they could prove that you knew that the ounce bar was not gold and that they were deceived into making you a substantial payment based on certain misrepresentations, then they could potentially sue you for some type of fraud.  The big catch, however, is what they can prove. 

Their first hurdle, as you mention, is the fact that they are the professionals that are supposed to be able to identify certain items.  It's one thing to defraud an inexperienced consumer -- but to defraud a company that regularly trades in gold and other precious metals is going to be a hard burden for them to overcome.  One of the ways they could overcome this is to show that you went to active lengths to deceive them by producing fake certificates or something.  If you simply took them a bar and said it's gold-- then they won't get far.

The second issue, as you note, is that they made a trade.  More than likely they have an "all sales final" sign posted somewhere--- and that works both ways unless they have something in writing otherwise.  Some will have you sign and mini contract which essentially requires repayment if the item you produced turns out to be stolen or not of the same character.  If you didn't sign something like this, their options are limited to the fraud options listed above.  If you did sign this type of document, then they could file a breach of contract suit against you.

If you are pushed on this issue, you need to get more info and how and when they discovered their mistake.  It is a bit odd that it took them a while to reach this conclusion.  Considering, again, that they are the experts, it should not have taken them long to learn of their mistake.  If you had time to pay bills with the money-- they had time to verify that the bar was gold. There is a possibility that they are just scamming you to get some of their money back. 

As far as your rights in this situation....

If there is something in writing that says you have to repay the different (as discussed above), then you should make payment arrangements to repay the amount.   If you are not contractually obligated and you did not intentionally deceive them, then you are not legally obligated to do anything for them.  The sale was final.  If you had found out the next day that the same bar was worth double the price you received from them, you would not be able to obtain a "do over."  Until there is a court order requiring you to pay them, you are not required to do so.  If they continue to harass you or do send you any notice of legal action, then visit with a consumer law attorney to file a response for you.  Considering that they are the experts, you would have a good defense to any fraud claims.

If you had no idea that the ouce of bar that you pawned or sold was not a full ounce of gold when you did the transaction then you did nothing improper. The mistake is on the pawn shop on the transaction that it did and it bears the burden for any loss it may have just like the program "Pawn Stars".