I believe that if a pastor is doing his job, there is going to be a burden that he must carry, and nobody else really understands the weight of it.
The greatest burden for a pastor to carry is not the amount of studying that he must do to prepare for his sermons every week. Studying is a delight and a joy. Preaching in the pulpit is hardly a weight either. The pulpit is the pastor's cockpit, he feels comfortable and at home behind the desk.
The administration of all the ministries; overseeing the people, training the leaders, and making sure that everything is running smoothly is not the burden. It might be a lot of work, but it is satisfying and even fun to see the people working together and accomplishing something that was a mere vision in the pastors mind.
Counseling people who are dealing with problems may be closer to the real burden, but it's not quite it. At least they're willing to meet with you to hear what the Bible has to say about their problem and are somewhat willing to do something about it.
The real burden that the pastor carries that nobody else seems to see or understand is that he has an overwhelming concern and care for the souls of others when, for the most part, the others have little or no concern for the souls of others nor do they have little or any concern about their own souls! The pastor stays awake at night before he can finally fall asleep thinking about the teenager who's is on the wrong path. The pastor wakes up thinking about what happened to the couple who is having marriage problems during the last 24 hours. The pastor constantly thinks about why so and so was absent from church for two weeks in a row. The pastor can't fully enjoy a good day at church when he looks at the pews and counts the number of members who should have been there but were not. If the pastor is preaching the Word of God faithfully, then he tries to deliver messages that he knows are going to help the spiritual health of his flock and then realizes that most of the people who would have benefited the most from it were not there because they don't care about the spiritual condition of their own souls!
We wish that we could shake them and awaken them from spiritual slumber but to do so usually only makes them angry at you - the one who cares for them the most. When a member leaves the church on bad terms, or because of un-resolved sin, you still think about them when you see a car on the road that looks like theirs and grieve in your heart that they may have made a life destroying decision. This, I believe is the greatest weight that the pastor carries 24 hours a day 7 days a week, even when he's on vacation or on his day off. Pray for your spiritual leaders and hold their arms up when they get weary. If they are good under shepherds, they love you very much.