Hi! New to this place but I love pondering about this sort of question.
Thing is, apart from what has already been said about the concept of "crime" in relation to anarchism, I feel that rape and serial killing are things that would not happen (or happen very little) in an anarchy.
Rape is not about sex. It is about power, domination, humiliation. It is about using sex as a tool to humiliate another person. I really don't see why a person would feel the need to do that unless they had been born and bred into a competitive environment.
As for psychopaths, there is much discussion about what causes their illness, but it is nowadays largely known that most of them are not killers. I share the belief that the environment in which they grow up is determinant to whether or not they will be dangerous to other people or to the society as a whole.
And even if there were murders - cold blooded, intentional murders - the people would view them differently from the way we view them today. Because the very idea of retribution, of wrongs making rights by being put together would not exist - or, in the very least not be as strong - in a cooperative society, I believe.
As to the ways in which an anarchist society would deal with a behavior that endangers its very existence or the safety or well-being of its components I think that yes, as has already been said above, there could be any number of ways in which that problem could be solved, exactly because there would not be a one institutional sistematized way to deal with it.
However, believing that an anarchist society is much more than just the external changes (no government, no property, no money), I find it hard to imagine that torture, execution or even imprisonment would actually take place.