First I was disturbed by Armen Alchian's below pronouncements; I thought them amoral or cynically indifferent. Later, I came to appreciate their depth; in fact, it turns out they hold the key to an answer long sought for by me:
Why has the state been part and parcel in all stages of the ongoing progress of human civilization, from the earliest Egyptian empires to today's powerful nation states?
The part of the Alchian interview that I am interested in starts at time mark 05:05:
Interviewer: Is there a conflict between your property rights and my political rights?
Alchian: No. I don't see any conflict. You have the right to go to talk to anybody you would like to talk to. To persuade him to commit suicide, to persuade him to marry your daughter, to persuade me to give you my money; persuade him to vote, persuade him to influence the king [...], if you want to persuade him to take away my property rights; you've got a right to do that. After all, property rights are what the rest of society will enforce for me. If the rest of society is not going to enforce property rights anymore, okay, we don't have them, they're gone.
Now, what has this got to do with the ubiquity of the state in the progress of civilization?
The problem with libertarians and anarcho-capitalists who demand the abolition of the state is that they do not appreciate that neither markets nor more generally liberty are capable of creating their own preconditions.
These preconditions are of a political nature. In Alchian's words, you have to incessantly "persuade the rest of society to enforce [certain] rights ... if they don`t ... we don't have them ... they're gone."
These rights have to be negotiated, fought for and established outside of the markets and within a framework that is not identical with and does enclose liberty only if you are lucky and politically/persuasively successful.
And this political process is an ongoing one. It does not stop at any point. Not even if a libertarian dictator prohibited that process, it would reassert itself sooner or later.
People are resourceful, imaginative and very diverse. This creates all the time any number of new and diverging ideas, interests and ambitions. Among these ideas, interests and ambitions there are many that drive people to seek to efficaciously influence other human beings and society at large. One avenue that gets you to exercise this influence is the political process. The most effective use of the political process allows you to access the most powerful instrument for impacting other people and society at large - and that is the state.
Hence, there will always be a tremendous demand for that instrument: the state.
If you drop out of the competition for state leverage of your concerns, others will overtake you gladly and mercilessly, and lever their goals ( and send you back to "square one"). A free society cannot stop that ongoing process, and ought not to. And hence, a free society is always prone to see interests and ideological proclivities ascend that are detrimental to liberty, just like an unfree society may see interests and ideological proclivities rise (and at some point eclipse again) that are favourable to freedom.
There is no totally reliable way known so far to control these swings (in favour of more rather than less liberty). Freedom is even more endangered, if out of disdain for the state you do not even make an effort to create political structures that protect liberty, but piously hope for private arrangements to do the trick -- after all the Wild West's cowboys could do without the state and some ancient Icelandic communities, too.
It is this core content of the demand for politics-and-the-state that explains the persistence of the state.
Whatever the merits and demerits, relative efficiencies and inefficiencies of the state, say, as transaction-costs-reducer and scale-economy-provider of protection, punishment and rights enforcement, it is this influence-leveraging component in the vast package of state services that makes it indispenasable and always in demand by the most active and ambitious in society.
See also More Thoughts on the State, and links contained therein.