Story telling is an art, but it is also a science. Just as there are immutable laws of physics that cannot be ignored, creative writing has rules which must be obeyed, a formula that must be followed. If readers and movie goers were conscious of this (which, fortunately, most are not), they might well feel insulted, even used, to know their emotions are being so carefully manipulated. Yet, if the writer violates one of these unspoken rules, the reader will instinctively know something went wrong and be let down by the result.
The protagonist must be sympathetic, or the reader will never care what happens to him/her. The plot must have conflict, or there is no story. The manuscript/screenplay must hit certain plot points, and the story must be brought to a satisfying conclusion. Fail to meet these (and many other) requirements, and the project will fall flat.
Using a tried-and-true framework is enormously helpful to a writer. It is a fool-proof safety net. Stay within the guidelines, and your story will work. The problem is we creative types like to think we're producing something original. We cry out in protest, "I refuse to prostitute myself by pandering to the masses, to sacrifice my art for the sake of a set of arbitrary rules." Okay, so that's a little overly dramatic, and the rules aren't at all arbitrary.
The fact is, there are no new stories, only new ways of retelling the old ones. Something in our human psyche longs to experience, again and again, the hero triumphing against all odds, love finding a way, and the bad guys getting what's coming to them in the end. These themes don't always prove true in real life, but we insist they be true in our entertainment. Otherwise we, as consumers, feel cheated and betrayed. That's one reason why departing too far from the well-trodden path rarely pays off for the writer. There's a standing joke in the industry that typifies this concept. What do publishers/film producers want? They want a proven commodity presented in a fresh way. In other words, "Give me the same thing, only different." Sure, no problem.
