I have always been a fairly driven, structured time-manager. I like having a schedule. Juggling the demands of a large family and a growing church is challenging. That's why you need boundaries. Those little time killers that eat away at your schedule - you need to control those as much as you can. Why?
1) Your stress level, for one. There are only so many hours in a day and when study time or visitation is compromised, you begin to panic. You get slack in your study and cannot feed your congregation or pray for them as you should. If you already battle flare-ups of temper OR depression, the best way to extinguish those character killers is to reduce stress.
2) Another reason to control those non-scheduled drop-bys is that it creates a neediness in your people - the unhealthy expectation that they deserve more of your time than something or someone else. Your accessibility has to be guarded because they will just drop by to have coffee for no particular reason. Though some of that is good and you want your people to love you and to enjoy your company, sometimes it boarders on the ridiculous because they have nothing else to do and they want you to entertain them.
3) The primary reason you should limit your availability is for your family. Your stress level, your time - these things directly impact them. Some pastors neglect their families, thinking, "They'll understand I am doing the Lord's work..." When in reality, they end up resenting the ministry and rebelling against your leadership.
How can you limit this? What can you do that will not be offensive but will still accomplish your objective?
1) Teach your people to protect your time. Subtle and not so subtle statements like, "I need a quiet office from 8 to 12 everyday for study, prayer, planning, administration, etc. If you need me, unless it is an emergency, call after 12." "Monday is my off-day. I need it to recover from Sunday. It is my "Sabbath" so please understand that I won't be in the office on Monday." "Friday night is 'Family Night.' We like to keep our family healthy by doing things together on THAT night. Unless your house is on fire or you have some emergency beyond needing me to crank your tiller (yes, that happened to me before) I am unavailable."
Some people will get the message immediately. Those are the folks who probably wouldn't trouble you anyway. Others will have to be reminded from time to time because they think you are talking to everyone BUT them. You have to teach them, pastor. And if you can't, teach your deacons to do it for you.
2) When you are busy and have that drop in, limit the time to 15 minutes. If it feels it will go beyond that, tell them you have to go make a visit that you had planned for that afternoon and see if they will go with you. You can fellowship in the car on your way to do ministry together. Chances are they will just leave at that point. But if they want to go with you, you have accomplished three goals: a visit, mentoring a member in visitation,and protecting your schedule.
3) Stand up after 15 minutes and tell them you have got to get back to your study but you want to pray for them, that God would do a great work in their life. Then walk them to the door. Sometimes we just need a little courage, men. If you have the luxury of having a church secretary, teach him/her to run interference for you.
Remember - you need boundaries in your relationships. Hopefully these few suggestions have been helpful to you...