Are child custody modification papers legal if they were not signed in from of a notary or witness?

Requirements may vary from state to state, but generally the modification of child custody documents doesn't need to be signed in the presence of a witness or notary.

Since you signed under duress, that would be an argument in your favor for invalidating the child custody modification.  Changed circumstances are grounds for modification of child custody.  The required documents for your modification of child custody vary from state to state.  Your court clerk can provide you with th documents you will need for another modification of child custody.  Generally, you will need an Order to Show Cause (court form) requesting a modification of child custody.  You will also need a supporting declaration signed under penalty of perjury stating the facts in support of your request for a modification of child custody.  File these documents along with a proof of service with the court.  When you file the Order to Show Cause, the court will schedule a hearing on this issue.  The proof of service verifies the date of mailing a copy of your documents to your ex.  You can use a court form proof of service or you can write your own.  If you write your own proof of service, it just says that you are over eighteen and the attached documents were sent via first class mail unless stated otherwise to ___________ (name and address of your ex) on _________ (date).  You sign and date at the bottom.  The date you sign should be the same as the date of mailing and the same date you file your documents with the court.