Stefan Molyneux (who puts out a popular podcast on this stuff) wrote an LRC article on stateless prisons a few days ago. In my carefree days I would've written a lengthy reply, but time is short so a few quick points:
(1) Why do libertarian males always use rape in the thought experiment??
(2) It seems that Stefan, in his desire to show that criminals would get punished a plenty in an-cap world, goes too far. He has the rapist being fined $500,000 and serving 5 years in prison, and getting all his utilities shut off, etc. In my view, the private judges deliver a verdict against you (which would almost always be a huge fine), and then if you don't pay all of the ostracism kicks in.
(3) To see it differently, even if the system were initially the way SM thinks, wouldn't most crime victims say, "Hey, rather than you serving 5 years, why don't you just give me an additional $300,000 and I'll drop the charges"?