Honestly. Wind. I hate it. Always have always will. Not just during hunting season although thats when my hatred for it becomes full fledged. Always messin up your stands then its finally right and you get there it decides to change. Then I have one stand that it doesn't even matter what way the wind blows. If its blowing anything besides straight out of the west it just swirls and blows all 4 directions all the time. I have lived in KS for 18yrs now and still hate the wind. Thought I would be used to it by now.
My number one is not finding an animal you put an arrow through. That sick feeling in your stomach is the absolute worst. Had it happen twice to me, and that is two times too many.
Not seeing any deer period. Spending all the off season with cameras, scrapes, scouting, overall preparation to not see the deer where you have seen them hundreds of times.
Top 2 in no order Walking through the woods in the dark.... I do it religiously, but I still don't like it. Wind