Frye doesn't seem as enamored with Hamlet as he is other plays (He calls the "to be or not to be" speech "hackneyed"). But he does recognize its importance:
"No other play has explored the paradoxes of action and thinking about action so deeply, but because it did explore them, literature ever since has been immeasurably deepened and made bolder."
His most fascinating insight is something I never realized before, which is that Hamlet is really a play with "three concentric rings of revenge tragedies."
The standard formula for a revenge tragedy needs three characters, Frye says. "A character to be killed, a character to kill him, and an avenger to kill the killer." Two of Shakespeare's other revenge tragedies, Julius Caesar and Macbeth, both follow this formula. Caesar is murder by Cassius and Brutus and avenged by Octavian and Mark Antony. Duncan is murdered by Macbeth and avenged by Malcolm and Mcduff.
In Hamlet there are three of these loops, all playing out at different speeds. At the center is the murder of Polonius by Hamlet, which Laertes will avenge. Around that is the main action of the play, which is the murder of Hamlet senior by Claudius, which Hamlet must avenge. And around that is the background story of Fortinbras senior who was killed by Hamlet senior in a duel on the day that young Hamlet was born. The Fortinbras we meet in the play is on his way to avenge that murder.
Three widening circles of revenge tragedies. Frye says:
"Their total effect is to neutralize the sense of the restoring of moral balance that a revenge is supposed to give us as a rule. Revenge does not complete anything, it merely counters something, and a second vengeance pattern will grow up in opposition to it."
This leads Frye to conclude:
"Hamlet seems to me a tragedy without a catharsis, a tragedy in which everything noble and heroic is smothered under ferocious revenge codes, treachery, spying, and the consequences of weak actions by broken wills."
In that way the play is perhaps truer to the essence of revenge than most conventional revenge tragedies which do have a satisfying resolution, the ones where a 'man-on-fire' avenges some great injustice committed, and can retire in peace, and we can feel as though something's been salvaged.
But by the end of Hamlet we don't have that feeling at all. Fortinbras walks in on an absolute massacre, and Horatio, charged with telling the world what happened, declares:
"So shall you hear
Of carnal, bloody, and unnatural acts,
Of accidental judgments, casual slaughters,
Of deaths put on by cunning and forced cause,
And, in this upshot, purposes mistook
Fall'n on th' inventors' heads." (V.ii.394-99)