No matter what the Kinks say, it’s not a good rule for writers unless you’re writing on a contract. If you’re writing for your own pleasure, or writing stories you want to tell, at least, then mostly you need to keep in mind that you’re the one you have to please.
However, it’s not a bad idea to keep in mind what your target audience wants. In general, the ideal situation is that what’s interesting to you is also going to be interesting to them, so you can happily scribble down all the stuff you like, and they’ll be just as satisfied.
There are times, of course, when opinions diverge. They, for instance, want the same characters in the same stories (thinly disguised as requests for sequels); you want to explore new ground. Or maybe you want to write the same characters, but moving on to new problems that make them less the characters your readers liked. There may also be times–and these are the tough ones–where you’ll know that a scene doesn’t belong in your story from a structural standpoint, but you’ll also know that the people who like your work will really like that scene.
Say, for example, you have a character whose dry, cynical wit is what you (and others) love about her. You’ve got her in this scene at a restaurant and have come up with some absolutely terrific zingers for her to fire at the slovenly waiter about the underdone food. It really shows off her character and will make everyone smile. But alas, the restaurant scene does really nothing else for the purposes of your story.
So what do you do there? If it’s something extraneous that you really should cut to move the plot along, but it highlights a part of the character that you like, and your readers will like, do you keep it? Try to work it into the plot a little more? Rework some of the more critical plot scenes to fill the same niche?
My answer, and it varies by situation, is that I try to get that scene to be more integrated into the plot. Failing that, I’m inclined to leave it. It adds color to the story, and if it’s got enough entertainment value, people won’t mind that it doesn’t move things along. You’re making the story fun to read and fun to write, and in the end, that’s what matters.
When you can do all that AND make it integral to the plot AND have a significant message AND do it in a fresh new way … well, that’s what we’re working towards, isn’t it?