The real restriction on consensus is not how many people are involved, rather that the people involved have similar values. If some people value fun, community, and cooperation, social justice, and other people value personal gain, exploiting others, and having power, (for example if the capitalist investors in a city sat down with the people that lived in the city.) a consensus process would never work.
also, it would be very hard to work in a consensus process with people like the religious right who believe that their job is to destroy the planet so they can go to heaven faster.
Outside of a situation where people have some common ground on beliefs/values, consensus is not a useful process.