Mystery fiction became popular in the 19th century for several good reasons. The simplest explanation is that there could be no detective stories until there were detectives in real life. Duh.
There were no organized law enforcement systems until the formation of the Surete in Paris (1811), created by Eugene Francois Vidocq who was, incredibly enough, a notorious criminal. England's Robert Peel established Scotland Yard (the nickname given the Metropolitan Police Service because of its location) in 1829, lending his name to the "bobbies" who still patrol the streets of London. In America, it was Philadelphia that in 1833 formed the first police department in America.
While the administration of justice in these organizations was not always as prevalent as it might have been wished, it was far superior to the previous system, which generally relied on state-sponsored torture during the interrogation process.
Crime, and crime stories, have always been with us. Think of Cain and what he did to his brother. No detective was needed to solve that murder, of course, since the suspect list was limited. Literature is rife with murder, from Tales of the Arabian Nights (under its many different titles), Chaucer's Canterbury Tales, to such Shakespearian dramas as Hamlet, Julius Caesar, and Macbeth.
The creation of real-life detectives made it possible for Edgar Allan Poe to invent a fictional detective, C. Auguste Dupin, in "The Murders in the Rue Morgue" in 1841. However, just because a prototype existed, why did it achieve a level of popularity so profound that it has lasted into the 21st century?
There is a theory (brilliantly advanced by me in the introduction to my anthology, The Best American Mystery Stories of the 19th Century--and, oh, yes, by dozens of scholars before me) that the detective story was able to achieve success only when people gave up their absolute adherence to religion, a phenomenon that occurred in the 19th century. The notion is that we all have a sense of guilt impressed upon us at a young age and that it can only be relieved by a higher power: to wit, God, or one of his lieutenants. When the extraordinary power of religious devotion diminished, the door opened for a different agency to lessen our guilt and this took the form of a detective.
The analogy of religion and the detective story goes like this: There is a sin (murder), a victim, a high priest (the criminal) who must be destroyed by a higher power--the alternative to God--the detective. Individuals identify with the light and dark sides of themselves--the detective and the criminal--and seek absolution and redemption. Thus, the denouement of the mystery is the Day of Judgment when all is made clear, the soul is cleansed--and the criminal, through the omnipotent power of the detective, is caught and punished.
Perhaps, however, there is an even more fundamental explanation for the enduring success of mystery fiction, virtually all of which dramatizes one of the simplest and purest components of human existence and behavior: the battle between the forces of Good and those of Evil. God versus Satan. The killer versus the savior. The detective versus the criminal. Since the majority of civilized society prefers good to evil, a great pleasure or, at least, comfort, may be found in the mystery story, in which it is prevalent for righteousness to emerge triumphant.
In the real world, as we have come to know, justice does not always emerge triumphant. Good people often suffer, they are victimized, they are defeated. Bad people win, they get away with despicable behavior, they inflict injury, whether physical or psychological. I hate that. Maybe that's why I was so strongly drawn to detective stories, however subconsciously, so long ago--and still am.
The moral, then, must be this: Thank God for mystery fiction.
Otto Penzler is the editor of The Best American Mystery Stories of the 19th Century.
Also on The Huffington Post: