Here is the whole paragraph:
Do not confuse "duty" with what other people expect of you; they are utterly different.
Duty is a debt you owe to yourself to fulfill obligations you have assumed voluntarily. Paying that debt can entail anything from years of patient work to instant willingness to die. Difficult it may be, but the reward is self-respect.
But there is no reward at all for doing what other people expect of you, and to do so is not merely difficult, but impossible. It is easier to deal with a footpad [a thief] than it is to deal with a leech who wants "just a few minutes of your time, please - this won't take long"... Time is your total capital, and the minutes of your life are painfully few. If you allow yourself to fall into the vice of agreeing to such requests, they quickly snowball to the point where these parasites will use up 100 percent of your time - and squawk for more!
So learn to say no - and to be rude about it when necessary.
Otherwise you will not have time to carry out your duty, or to do your own work, and certainly no time for love and happiness. The termites will nibble away your life and leave none of it for you.
This rule does not mean that you must not do a favor for a friend, or even a stranger. But let the choice be yours. Don't do it because it is "expected" of you.
There you go. Think about it. May it helps you to have a better life.
More Heinlein quotes can be found here