A right away agreement

You say you had an agreement: did you give him anything in exchange for it, such as paying him a fee, paying road maintenance, giving up some other claim or cause of action you had against him, etc.? If you did, then if you gave him something of value (called "consideration" in the law) for the agreement, then the agreement was an enforceable contract. As long as you did your part--gave him whatever you had agreed to give him, etc.--he could not terminate the agreement without your agreement, or consent, to do so; and if he did, you could sue him for "breach of contract" to enforce the agreement. 
However, if you did not give him something of value for his agreement, it was not a contract: it was just a "gratuitious" (freely made) promise, and such are not legally enforceable--someone can promise you something then change his/her mind. A way to think of it is this: you don't get something for nothing, so if you provided nothing for the use of the road, then you are not entitled to anything--you only have the use so long as he voluntarily agrees to give it to you.
Now, there is a legal mechansim, called a "prescriptive easement" or "easement by prescription," to have a court order that you get access across another's land or road if there is literally *no other* way to get to your land--you are "landlocked," as it is often called. But if there is another, even if much less desirable, way to access (or build your own access road to) your property, you need to take that. And even if there is no other way to get to your property and you would be granted a prescriptive easement, it can be time consumming and expensive to litigate (go to court) to get one; you are likely better off to work it out with your neighbor, such as by agreeing to pay him an access fee or take over road maintenance for him.
If you have to seek a prescriptive easement due to their being no other way to access your property, consult with a real estate attorney about doing this.