At the outset, let’s point out that a great many cartoons do end up scaring kids. There has long been criticism of classic Disney films for scaring children. I don’t accept the criticism because dealing with things which scare you is part of growing up and learning to separate real fears from fake fears and deal with all of our fears appropriately is a necessary part of life. But that is neither here nor there. For while those films may scare children, scaring children is easy as they haven’t yet fully grasped the difference between the real world and make believe. Scaring adults is different and those films don’t scare adults.
So the next question is has there ever been a truly scary cartoon? I can’t honestly think of one. Cartoons are generally aimed at kids and, thus, tend not to really try to scare very much. And those that are aimed at adults tend either to be violent and/or sexual, and aren’t really aimed at producing emotions so much as they are aimed at seeming “cool.”
Lord of the Rings as an animated film. This film was largely done with live action which was then rotoscoped into appearing animated. Then main characters were hand drawn like normal animation and added to the action. The result was a dark and effective film which gave you a lot of tension and a lot of believability. In fact, in some ways, this film was stronger than the Jackson films (Jackson, by the way, initially denied having ever seen the 1978 film, but later admitted that he had and then claimed that some of the things he stole from that film were an “homage”... an homage to a film he repeatedly denied ever seeing). This again suggests that we can create fear.
So what would it take to make an effective animated horror film? Well, for one thing, I think you would need the animation to be more realistic and life-like and less cartoon-like than normal because you need to strip away the safety that comes with being able to laugh at the characters because of the way they are drawn. I think you also need to avoid anime or the 1950's comic book style, both of which scream “animated.” Similarly, you need to avoid the other excesses that get nerds everywhere to go “cool!”, like splattering blood and strange effects. In effect, you need to avoid anything that screams: “animated!”
Secondly, you would need a strong character-story. This would be essential because that is what pulls you into the story: caring about the characters. It’s no coincidence that most of the best horror movies involve strong character stories.
Third, I think you would need to go for menace and tense rather than outright horror. In effect, you would need to scare the audience throughout as we discussed with Paranormal Activity rather than lining up a complicated plot with a single payoff. It would also be wise to restrain yourself from the desire to draw creepy monsters. Monsters almost always end up looking silly and injuring horror films, and adding a cartoon monster would probably make that doubly so... stick with eyes and shadows.
So what do you think? Do you think it can be done? Any other conditions you would add? Any conditions you would change? Any scary cartoons you can name?