You've got bigger balls than I do if you're going to go to some dudes house that's been giving you crap about hunting, who's already mad at you, knock on his door and try to tell/show him he's wrong.
I'm still laughing! lmao That's just what I was thinking. These days you have NO IDEA who you're dealing with. I do know you don't want to go on that dude's property for any reason.
I wouldn't go to his door but I'd damn sure put him in his place rather than take a cussing or being constantly harassed.
If the posted signs are on your lease, just take them down and place them on the other side of the tree towards his property. Then when he comes to investigate, he will be the one trespassing on posted property. Make sure to point it out to him when he does.
My reply earlier was based on being a property owner not a guest hunter. I understand that as a guy with permission is different as an owner. I would piss pound any jackwagon that tries to bully his agenda on my property, then again the neighbor seems to know the situation in the OP's case.