Levels of probability

July 26th, 2010 by madhadron

A mechanical ant acting as a benefactor of mankind.

literaure on probability and statistics is littered with the rancor between different camps of probabilists and statisicians. The frequentists see a probability as the idealized limit of repeating a given trial infinitely many times. The Bayesians insist that probability describes the strength of beliefs.

Pure mathematicians who practice Erdos magic form a third camp completely isolated from the first two. They take a class containing their prey, and seek a probability distribution on that class which assigns their prey a nonzero probability, which is a nonconstructive statement that their prey exists. They are obviously unconcerned by any statistical interpretation of probability.

All three camps use Kolmogorov’s formulation of probability, but this isn’t the only possible formulation. Feynman followed a comment of Dirac to formulate quantum mechanics with negative probabilities. Nelson’s Radically Elementary Probability Theory starts from random variables in nonstandard analysis, and builds a limpid version of the modern machinery of stochastic processes (I didn’t understand martingales until I read his book). Some work on quantum gravity (or see the papers) eschews real and complex numbers, and builds probabilities in far more exotic topoi.

What vague structure do these share to make us call them “probability?” There is some concept of an event – the result of a coin flip, the existence of a mathematical object, or the tunneling into existence of a universe – which is built into some lattice-like structure expressing combinations of events. There is some measurement of probability. Conventionally, it is interval from 0 to 1, but could be any ordered set. Finally, there is some mapping — I won’t say function, as it may not be — between the lattice-like events and the probabilities.

This is so vague that we could hand wave almost any pair of monoids with a relation between them into a “theory of probability.” What prevents us from doing so?

Probability began with games of chance. Every formulation I mentioned above can be restricted to that domain, and their restricted forms are identical. The differences arise when the theory is stretched beyond a game of poker. These games of chance are the ur-theory of probability. It is not a set of grand underlying principles, but a set of rather prosaic examples. Any “theory of probability” should handle them conventionally.

The prosaic ur-form is ubiquitous in mathematics. “Number” began with 1, 2, 3, etc., and any set of numbers today is expected to contain them. Functions began with polynomials, which continue as functions no matter how the definition is stretched.

It’s odd, isn’t it, how the simple cases thunder down the years and shape mathematics?

Why I don’t trust the cloud, Part 2

July 19th, 2010 by madhadron

A mechanical ant acting as a benefactor of mankind.

In my last post I posed the problem of creating a robust network for long term storage, a “cloud,” in analogy with the robust communication network that we have in the Internet. This post sketches the basic ideas of such a cloud, starting with the atomic unit of storage, a blob, and describing how it is passed from machine to machine.

The basic unit of storage in a cloud is a blob of binary data with a unique identifier, plus some kind of header to handle permissions and capabilities. The header is beyond the scope of this sketch, though, and I’ll ignore it.

What shall the identifier be? A long string of bits is the obvious answer. How long? If we make it long enough to have no more than a 10% chance of collision if we assigned a unique identifier to every atom in the universe, that should be long enough. Assume the number of atoms in the universe to be 1080, and assume that the number assigned an identical label is Poisson distributed. Then we need 270 bits, or just under 34 bytes, to have unique labels.

I will steal an idea from git and say that the identifier should include a cryptographically secure hash of the contents of the blob. This guarantees that a blob can be fetched and used with confidence. If the contents were tampered with, either the checksum no longer matches, or the blob has a different identifier.

Finding space for extremely large blobs may be difficult. Most hosts can squeeze in another 10MB blob. 10GB is rather harder, and 10TB requires special provisions. Instead of trying to pass around such an enormous blob, it should be broken into manageable shards, each a blob unto itself. Then an index blob containing a data structure like a B-tree with the blob identifiers of the shards at its leaves acts as a proxy for the whole mass of data.

In contrast to the carefully designed identifier and header of a blob, its contents are completely unprotected. Just like connections on the Internet, the contents should be encrypted when desired at another layer.

A blob enters the cloud when a host receives and accepts a store request for it. Then the blob is transferred to the host, which stores it and sends off to hosts upon request. If the host can’t store it, it rejects the request and passes it along to other hosts around it until some host accepts it.

If a host needs to clear space, it can send store requests for some of its blobs to other hosts, transfer the blobs to them, and delete them locally, though it should continue to redirect requests for the blobs to their new hosts for a time.

Of course, this picture is oversimplified. A few minutes thought reveal a number of other features such a protocol would need to be practical. One copy of a blob is not enough. The number of copies should always be high enough and well enough distributed to protect against loss when hosts die, and should change with the recent demand for the blob. For example, a host which is receiving a disproportionate number of requests for a particular blob should start sending store requests to other hosts and then redirecting some of its requests to them.

Also, this picture seems to make the cloud a free for all. Why would I put storage space in the cloud if all my local storage is going to be filled by random blobs and I have to wait for my blobs to wend their way in from the other side of the world? To solve this, blob may have a home address, and the cloud should conspire to keep at least one copy of the blob close to it, though not perhaps on it or the home address can easily end up thrashing, trying to offload blobs while having them sent back to it again.

Similarly, accepting store requests and responding to retrieval requests needs to be controllable. Strict, rule based filtering like Linux’s iptables is not really practical. The quality of service agreements naturally take the form of statistical promises such as “when these local servers have recently been sending lots of store requests and the disk is above a certain fraction full, begin denying external store requests and start offloading blobs whose home address is not here.”

Finally, storing and retrieving blobs is a very sparse set of commands. At the very least, it should be possible to delete them as well. Allowing the owner of a blob to delete it while preventing others from doing so is a very tricky problem, far beyond the scope of this sketch.

The semantics of the cloud are technical and subtle, but the basic idea is simple: a blob has an identifier, is sent out, and multiple copies of it are balanced among hosts. The number of copies adapts to the recent demand for it, and the cloud shuffles blobs around to keep itself functional. A simple mental model will suffice for most of its users. These protocols aren’t enough, though. To make the cloud truly useful we need a lookup system, analogous to DNS for looking up hosts. We’ll examine what properties it must have in the next post.

Why I don’t trust the cloud, Part 1

July 11th, 2010 by madhadron

I don’t trust the cloud. I’m not being paranoid about a white fluffy thing in the sky over me. I mean the mass of connected services, storage, and processing power on the Internet which have been labelled “the cloud.” I have finally figure out why.

First, a digression on network architecture. On the Internet, if you can connect to a host, in the future it is a network failure unless one of two things is true: you can connect to the host, or the host is no longer part of the Internet. If a plug gets pulled somewhere along the way, either by accident or design, unless it is the plug to the host itself, it is a network failure if you cannot access it. Let us call this property “robustness.”

Robustness is obviously desirable. I can connect my hosts to the Internet and know that people connecting to them depends only on my and those connecting, not the whims of any carrier in between. This lets me have confidence that five, fifty, five hundred years down the road, if my host is still attached to the Internet and so is theirs, they can still connect to each other. Of course, there may be lag as theirs has moved halfway across the solar system, but that is neither here nor there.

Robust networks also make effective censorship impossible. The only agent which has managed even partial censorship is China, and they are far from successful. This horrified movie, television, and music companies when they came online, since their whole existance was based on digital patterns existing in certain places and not in others.

Given this, why do robust networks exist at all? They seem far too disruptive to be of value. There is a simple reason. The value of a communications network increases with the number of people on it, and robust networks, because they can be used confidently by all parties, grow far beyond what is possible for nonrobust networks. The largest nonrobust networks are things like a telephone company’s network or a cable provider’s network. These are dwarfed by the Internet, just as all proprietary two way radio systems are dwarfed by CB and ham. The Internet is thus vastly more valuable to its users than a nonrobust network. The only parties for whom it is not more valuable are those who extract value by some form of censorship.

Now let us look at the cloud. There are lots of clouds, actually. There is Amazon’s EC2, and Google’s various offerings, and 37signals’ Backpack, and endless others that will store your data out there in the ether somewhere. Clouds consist of two resources: computing power and storage. Can we define an analogue of network robustness?

Computing power is already robust in clouds because the underlying Internet is. By definition, computation is ephemeral, since if it doesn’t terminate, it’s not very useful. It is enough that I be able to reach a host again in the future. No promises need be made besides that for computation to be network robust.

Storage is a different question. By definition, we are asking for guarantees that we can access data in the future. None of the clouds mentioned above is robust since any one of them could tomorrow vanish or turn evil, taking our data with it.

So what do we demand of a cloud in order to be robust? We must be able to guarantee that in five, fifty, five hundred years the data we put into it must still be retrievable, private if we wished it to be, and intact. It seems like a tall order, but so was the Internet.

In my next post I’ll outline my ideas for how this can be accomplished.

Success = Power

July 6th, 2010 by Stormlark

How can genes be successful if they are not passed on?  If we take the selfish gene hypothesis to its conclusion, we can see rationales for certain types of altruism and even eusociality, but not refusal to breed.

Having fewer children, focusing on social status and achievement do not make sense in this rationale. Yes, more children survive.  Yes, life is more comfortable and longevity is positively correlated with high social status.  Your genes, however, don’t really care how much longer you live after you’re done dealing with children.  Unless you’re a caretaker for bodies that will pass on genes like yours (a grandmother, uncle etc.) you have failed at the fitness game.

Or have you?

Movies like “Idiocracy” make a great deal out of smart, successful people being out-bred by fecund dullards in the developed world.  The message appears to be that it is the fecund dullards who are evolutionarily fit while the smart people are being selected out.

I propose that we are witnessing two discrete methods of optimizing a slightly different type of fitness which I’ll call “success”:

Let us define the success of a gene (possibly a meme, but I’ll use “gene” in these arguments) as the amount of energy that its carriers can command at any given time.  This is, of course, somewhat different from the traditionally understood goal of maximizing a gene’s representation in a population.

In the absence of technology, social/command structures or access to them, maximizing a gene’s representation in a population is still the best way of maximizing the amount of energy expenditure (let’s call this “power”) at its command.  If you or your children are unlikely to be able to climb the social ladder or get control of lots of unsupervised power, your best bet is to breed.

Now, let us consider the slowly-breeding elite in a social, technological society.  If genetic success is measured by power, then highly energetic practices moderated by technology (science, commerce, travel, construction, manufacturing, war, art) and the capacity to control many who practice them (social/corporate status) start to outweigh breeding in certain circumstances, especially those in which breeding or resources are already constrained.  By having fewer offspring and concentrating resources on those few, it can be argued that the power accumulated by a gene increases far beyond what simple additional bodies would have added.

So, the questions become:

  • How much energy expenditure does a human command simply by virtue of being a human?
  • How much of a person’s energy do his controllers (bosses, leaders, etc.) command?
  • How much status / control of others outweighs breeding?
  • At what energy expenditure does technology outweigh breeding?  (i.e. Does this scale linearly with the energy of people.  Keep in mind that many of the controlled use technology themselves.)
  • Is this, rather, a meme, working at cross-purposes with the “genetic prime directive” which probably equilibrates roughly to “BABIES!”

Now, let us undertake a thought experiment with this new definition of success:  SENS is working on a cure for aging.  Suppose that, for some of us, this will mean hundreds or thousands of years of youthful health.  Let us also suppose that our potential “long-lived” are in an already saturated environment; that resources for population growth are limited or not available.  It seems clear that maintaining continuity in resources, bodies and social power will work well for those genotypes / memotypes concentrating on social/technological power rather than purely somatic power.

\therefore Immortal Overlords can be forgiven for being late bloomers when it comes to mate acquisition.

A speculative fiction reading list

July 5th, 2010 by madhadron

A mechanical ant acting as a benefactor of mankind.

This is a curated list of gems of speculative fiction. It is not a top 100 list, a comprehensive list, a beginner’s list, or any other such list. It is a curated list. It is short, brutally trimmed, and I explain why I chose the works on it.

I have omitted works which we all knew and loved as kids such as C.S. Lewis’s Narnia and Heinlein’s Between Planets or Citizen of the Galaxy. I have omitted works that where the writing drove me up the wall or the characters were utterly cardboard.

I have omitted works where the speculative conceit has not aged well. A great deal of speculative fiction was a collective effort to expand what we could think about familiarly. It succeeded, as a perusal of Edgar Rice Burroughs by a modern reader demonstrates. Today that literary effort is of historical interest, but the majority of it verges on unreadable.

So, ordered by author,

Isaac Asimov, Profession

Lois McMaster Bujold, The Curse of Chalion

Mikhail Bulgakov, Master and Margherita

Greg Egan, Dark Integers

Robert Heinlein, Time Enough for Love

Zenna Henderson, Ingathering: The Complete People Stories of Zenna Henderson

Frank Herbert, Dune

L.E. Modesitt, Gravity Dreams

Eric Frank Russell, And then there were none

James White, Sector General series (Beginning Operations, Alien Emergencies, Tales of Sector General, and the two that didn’t make it into the anthologies, The Genocidal Healer and Double Contact)

Stanley G. Weinbaum, The Best of Stanley G. Weinbaum

There it is. Seven books, three short stories, and a series.

What’s wrong with our thoughts?

June 29th, 2010 by madhadron

David Stove wrote an essay What is Wrong with Our Thoughts? that includes a list of 40 propositions about the number three, which are a small museum of the pathology of thought. He adds, “after the first two specimens, you will not know…in what way they have gone wrong.”

Actually, I do. They’re good case studies, so let’s go through them, organized by error.

Before diving in, let’s clarify what three is. We can write down formal languages, which are sets of sequences of symbols, called sentences. Any other structure is a model of a formal language if there is a mapping of the symbols onto elements of the structure such that all the sentences hold. This is part of a field of mathematics called model theory.

We can define a formal language of 0, \forall x : S.x \not= 0, and \forall x,y : S.x = S.y \equiv x = y. If we can find some set of objects in a structure, call them a, b, c, and d, such that S.a = b, S.b = c, S.c = d, and S.d is undefined or something else besides these four, then we are justified in calling a zero, b one, c two, and d three. In other words, three is any element of a model of the formal language given by the Peano axioms which satisfies S.S.S.0.

I’d like to pause and quote Dijkstra (EWD1123 – warning: PDF):

“We now take what is a standard step in mathematical theory building. The step is taken after the introduction of a notational novelty – such as a new abbreviation or a “mathematical macro” – for formulae that were interpreted in a familiar domain of discourse. The step consists of starting with a clean slate and axiomatizing afresh the manipulations of the new formulae. In so doing, one creates a new domain of discourse; the role of the old, familiar domain of discourse, that used to constitute the subject matter, is thereby reduced to that of providing a possible model for the newly postulated theory. It is essential that the axioms of the new theory – which can be interpreted as theorems in the old universe of discourse – are clearly postulated as such and that the new theory is derived from them without reference to the model of the old universe of discourse. This is the only way to assure that the axioms of the new theory provide an interface that is independent of the old universe of discourse and that, hence, the new theory is safely applicable to alternative models.”

This is the inverse of the description of model theory above. After axiomatizing, we find an object which we have labeled three. It is all to easy to confuse whether we are talking about three in the axiomatized formal language, three in this or that model, or the symbol “3″ that we write on paper, and many of the propositions fall into exactly this trap. Now let us turn to the propositions.

First, there is the statement of a classically false proposition.

1. Between 1960 and 1970 there were three US presidents named Johnson.
2. Between 1960 and 1970 there were three US presidents named Johnson, and it is not the case that between 1960 and 1970 there were three US presidents named Johnson.
9. There is no number three.
10. Three is the only number.
14. The sum of three and two is a little greater than eight.
29. Three is a positive integer, and the probability of a positive integer being even is 1/2, so the probability of three being even is 1/2.
33. How is the addition of numbers possible? Nothing can make the number three into four, for example. (Unless he means the object three in a model be the object four in another convention, which is just fine. Let 1 for one model of the integers be zero in another way of making the structure a model of the formal language.)
38. The unconscious significance of the number three is invariably phallic, nasal, and patriarchal. (I hesitated for a moment about this one, but then I thought, “boobies!”)

(1) would be fine if you were writing an alternate history story. (2) is false in any structure anyone would call a logic. (9) is a statement that there is no model corresponding to the formal language. (14) is again logically wrong given the definition of three.

Then there is material that is simply undefined.

3. God is three persons in one substance, and one of these persons is Jesus, which is the lamb that was slain even from the foundations of the world. (So Jesus is one personality of someone with split personality disorder and…)
26. The tie which unites the number three to its properties (such as primeness) is inexplicable. (If he means a predicate, it’s easily explicable; if he’s trying to be profound, what does he mean by a tie?)
30. In some previous state of our existence we knew the number three face-to-face, as it is in itself, and by some kind of union with it. (I don’t kiss and tell…or whatever it is one does to a number and tell.)
35. We get the concept of three only through the transcendental unity of our intuitions as being successive in time. (Is this the same kind of union as in 30? Is someone a little kinky?)
36. One is identity; two is difference; three is the identity of, and difference between, identity and difference. (You defined identity and difference as one and two; now define identity of and difference between. This could also just be 3=1 and 3=2 and 2-1=3, all of which are false.)

There is the confusion between the label “three” and the object in a model which is denoted by three. If we speak of the label “three,” then this is true:

7. Three lies between two and four only by a convention which mathematicians have adopted.

If we speak of the object we point to, and we use the usual ordering on natural numbers, then the statement is logically false. A convention is a choice among several ways in which a structure is a model for a formal language. Left-handed or right-handed coordinate systems are a convention. Ordering in the integers is not.

The same confusion taken to an extreme is seen here:

8. There is an integer between two and four, but it is not three, and its true name and nature are not to be revealed.

If we refer to three, the name of a symbol, then call it whatever you like, but until you and your cult get around to inducting me, I shall go on calling it three. If we refer to the element corresponding to the term in our formal language, then the statement that it is not three is logically false.

Similarly, the following statements are all false because three lies between two and four by construction. If an element in a particular model did not satisfy this, it would not be called three.

4. Three lies between two and four only by a particular act of the Divine Will.
5. Three lies between two and four by a moral and spiritual necessity inherent in the nature of numbers.
6. Three lies between two and four by a natural and physical necessity inherent in the nature of numbers.

And then there is the idea that three is a unique thing, rather than anything that we care to so label that satisfies our fragment of a formal language, though some verge into undefined.

16. Three is a real material object.
17. Three is a real spiritual object.
18. Three is an incomplete object, only now coming into existence.
19. Three is not an object at all, but an essence; not a thing, but a thought; not a particular, but a universal.
20. Three is a universal all right, but it exists only, and it exists fully, in each actual triple.
21. Actual triples possess threeness only contingently, approximately, and changeably, but three itself possesses threeness necessarily, exactly, and immutably.
22. The number three is only a mental construct after all, a convenience of thought.
15. Three is a real object all right: you are not thinking of nothing when you think of three.
24. The number three is that whole of which the parts are all and only the actual inscriptions of the numerals, ‘three’ or ‘3′.
27. The number three is nothing more than the sum of its properties and relations.
32. Since the properties of three are intelligible, and intelligibles can exist only in the intellect, the properties of three exist only in the intellect.
34. What the number three is in itself, as distinct from the phenomena which it produces in our minds, we can, of course, never know.
37. The number three is not an ideal object of intellectual contemplation, but a concrete product of human praxis.
39. The three members of any triple, being distinct from and merely related to one another, would fall helplessly asunder, if there were not some deeper non-relational unity of which their being three is only an appearance.
40. It may be – though I don’t really believe in modalities – that in some other galaxies the sum of three and two is not five, or indeed is neither five nor not five. (Don’t laugh! They laughed at Christopher Columbus, you know, and at Copernicus; and even the logical law of excluded middle is being questioned nowadays by some of the sharper young physicists.)

Some of the statements are experimentally false.

13. Three is a lucky number. (i.e., in all random situations where you have a choice of numbers, your average payoff will be greater if you choose three.)
28. The number three is neither an idle Platonic universal, nor a blank Lockean substratum; it is a concrete and specific energy in things, and can be detected at work in such observable processes as combustion. (There is no form of energy which it is sensible to call three.)

Some of the statements can be true, depending on context. If you are working in a very restricted set where only the fragment of the formal language up to three has a model, then this is true:

11 Three is the highest number.

And if you’re talking about the number of times you have to die before someone will forgive you, then this also may very well be true:

12 Three is a large number. (Is this the number of times you have to die before someone will forgive you? Starts to look pretty big, then…)
31. How can I be absolutely sure that I am not the number three? (If you are the object corresponding to three in a model of the natural numbers, then you most certainly are the number three, as someone’s drill sergeant has doubtless informed him.)

I think the sense Stove meant is that they are everywhere true, that is, “for all contexts, three is a large number.” Then these are logically false.

This one makes the same mistake, but in a particularly odd manner:

25. Five is of the same substance as three, co-eternal with three, very three of three: it is only in their attributes that three and five are different.

If you have a model of the natural numbers where “substance” is defined, then it may be that they are of the same substance, and it may not. Or you may just not have any way of defining substance in a useful manner.

The computer scientists will note the immediate problem with this one:

23. The proposition that 3 is the fifth root of 243 is a tautology, just like ‘An oculist is an eye-doctor.’

There are two relevant definitions of tautology. One is used in formal logic, but since we are comparing to the English language, that isn’t the one we are interested in. The other is “redundant use of words.” Anyone who has to compute the fifth root of 243 will disagree that being told that it is three is redundant.

At least in this list, it’s very clear how all these thoughts go wrong. And the philosophers wonder why mathematicians and scientists largely ignore them?

Transistors After the End

June 28th, 2010 by Stormlark

A mechanical ant acting as a benefactor of mankind.

One day, it is going to happen.  A meteorite will strike.  The bombs will drop.  The polar caps will melt.  A hypervolcano will erupt.  Someday – we can’t know when – something very bad is going to happen to much of the Earth all at once.  The roaches might be fine, but how will we poor humans make out?

Provided I’m not vaporized outright, I intend to live well.

A life of scrabbling for the ever-dwindling fruits of industry’s last day is a romantic image, one many people seem to enjoy contemplating, but I have no taste for it.  I intend to live a life not much different from the one I live now, save for the Earth’s higher lava content and the drastically reduced population.

This will require a certain group of technologies that we already possess, like transistors, but it will also require a new meta-technology that is present only in its infancy.  We must be able to arrive at the products of our current industry from some small surviving fragment in a time span much shorter than a human lifetime.  The following is a sketch of my plan to enjoy transistors after the end.

  • Define the minimal set for unit manufacturing capability.

If we’re interested in being able to reproduce and regenerate our material culture after it is damaged, size, specifically smallness, is going to be important.  We can reduce the scope of our inquiry by considering capability instead of capacity.  For example, we needn’t worry about creating three cars per second with a car manufacturing infrastructure the size of the one we currently have.  Instead, we need only worry only about our capability to make cars at all (that even one car can be produced).  Let’s call the set of these unit production capabilities \emph{G} and the minimum infrastructure (tools, information, skills) required to support them \emph{H}.

\emph{G} is certainly much, much less than current industrial capacity for most products.  If we call our current production capacity \emph{M}, we can see that \emph{G} \subset \emph{M}.   Because technological products are used to create infrastructure, we can even see infrastructure as a product of itself.  My fear is that the size of \emph{H} right now approaches the size of all of global industry.  Simply put, I believe that, at this moment, we need almost the entire global infrastructure in order to preserve our ability to create even one of everything we make.

This is like saying that it takes almost every mushroom in the world working in concert in order to make even one more mushroom.  It is part of why massive destruction of industry is so terrifying: the fruits of current industry are only available if most of industry survives.

  • Construct a framework for mapping technological dependencies.

Anyone who has played an RTS knows that one of the most challenging parts of the game is managing technological dependencies.  As usual, Sid Meyer and Blizzard have done the hard work for us.  They’ve demonstrated how this constraint system can be understood, worked around and, for lack of a better term, played with.  Rendering the “tech tree” in these terms will allow ordinary people to comprehend what must be done in order to preserve their comforts and enable them to follow the steps of the tree in achieving them.

As \emph{G} \subset \emph{M}, we are likely to need to prune our current (\emph{M}-based) understanding of our tech tree if we want to make one for \emph{H} that is both complete and of minimal size.  Defining where we can cut corners in our tech tree should be the key to minimizing rebuilding or reproduction times.

  • Develop the smallest and most efficient possible \emph{H}, optimized for portability and democratization.

As we know, efficiency and smallness usually work at cross-purposes in an industrial setting.  This has to do with economies of scale and all that.  In order to make technology survivable, we will need to ride out our tendencies toward miniaturization, personalization and self-replication.  The RepRap is a good first start here, but we will need to take it much further in order to lose very little technology to disasters.

These are, of course, tall orders, but ones which we are much better equipped to tackle now with an advanced and existent technological infrastructure than in the ashes following World War III or a Zombie Apocalypse.  While the odds against a catastrophe occurring at any given point in time may be slim, there are two factors that outweigh this:

  • The cost of losing everything we have now is high.

A post apocalyptic Earth will not be some sort of pastoral Arcadia as many seem to believe.  We will lack things we’ve come to enjoy, like health-care, sanitation and teh interwebz.  Any backup expert will tell you that a plan to implement backups after an incident is just as important as having the backups themselves.  Without some plan, we will find it difficult to implement health-care, sanitation and that series of tubes back together again in time for our grandchildren to enjoy (and not die from lacking) them.

  • There is a great deal of time out there.

While being bitten by a shark is unlikely, if millions of people keep swimming in shark infested waters, eventually someone is likely to be bitten.  This holds true for epidemics, hypervolcanos, asteroids and possibly even chaotic population collapses.  Even if the probability is low on very short time scales, in the long run it approaches a certainty.

This topic will bear much further study and crowdsourcing.  We can ensure that humanity will be okay in the wake of almost anything, provided we take the steps now.  I know we can do it, Internet!

Look at the humans

June 21st, 2010 by madhadron

A mechanical ant acting as a benefactor of mankind.

Social engineering is the natural prelude to all other engineering. Let me begin with a case study to illustrate my point.

I am part of an IT group created by fiat. We are all members of other organizations scattered around Switzerland. The powers that be mandated that we become a cohesive human resource for a scientific enterprise they were funding, which is also spread throughout Switzerland.

One of the first things we had to do was to establish communication within the group. A few of us sat down to hash out how to do this. It was a group of fairly savvy programmers, who immediately began talking about the relative merits of various mailing lists and wikis and forums for communications within the group. There were proposals for an ontology of subject headers for the existing mailing list (which had carried a total of less than ten messages up to that point). There were ideas of integrating this program and that tool, of databases to store the details of various forms of communication.

None of it would have worked. Or rather, it would all have functioned. The programmers in question were quite good. But it would have failed miserably.

They hadn’t thought of the humans.

Our goal was an open channel of communication among a group of about twenty people scattered at half a dozen sites around a small country. By an open channel of communication I mean both the technical means to communicate, and the psychological impulse to do so.

Technical means abound today. Anyone can name half a dozen ways from Twitter to walking over and having a cup of tea with the person. The psychological impulse is the problem.

How many accounts do you have scattered around the Internet that you never check, that you don’t even remember? I’m sure I’ve lost track of at least fifty over the years. Why? The rate at which I received information I valued through them did not justify the effort to deal with them. This could be because there was no information, or because the information which was valuable to me was lost in more noise than I wanted to dig through.

With that in mind, there were three obvious criteria for our communication channel:

  1. The channel must require as little effort as possible to get information from.
  2. The channel must not be absolutely silent.
  3. The channel must not be so noisy that it eats the time of the listener to deal with it.

How did we fulfill these? (1) is easy. Use a mailing list. If you are a programmer and you are not Don Knuth, you probably check your email. Some of us only do so once or twice a day, only on workdays, and dream of Knuth’s paradise, but we still check it. Since everyone is already accessing the carrier of the channel anyway, the effort to get information from the channel is zero.

Since this was a two way channel among a closed group of people, (2) required getting those involved to actually send things to the mailing list. These were people working on disparate projects at disparate sites. Official business would probably have constituted one message every month or so. So I stood up in front of the whole group in a room and *personally* gave them permission to post anything they thought was might interest someone else on the list, whether it’s a popular science article or a new computer game. What authority did I have to do so? None whatsoever, but no one else did either, so I just snatched up the mantle and ran with it.

(3) solved itself in this context. Since someone in the group had to initiate the traffic, and there were only twenty people, all of whom had other things to do, we were unlikely ever to reach a point where there was too much traffic.

Criteria (2) and (3) are vague, though. How much is enough traffic? How much is too much? We thought about how we deal with our email, and how often we would have to receive a message from some channel in order to think of it readily as somewhere to communicate, and decided that anything less than two to four messages a week was a closed channel. Anything more than two a day would be a noisy channel.

Noisy channels are relatively easy to fix, though, provided it’s not all spam. The traffic usually divides into relatively disjoint hunks. The Haskell mailing lists are divided into an “announce” list and a “cafe” list, the first a low traffic list, the second a general discussion list. Large projects like Boost and PHP split into many mailing lists for various groups. More overlapping groups, such as Marc and Dianna MacYoung’s Animal List use subject headers and let each reader filter what he wants from the stream.

So in the end we had our mailing list, and a wiki that someone had already set up, and, as I said, I asked everyone to send things they found interesting to the list along with official business. And what do you know? Not a couple of weeks later I was involved in a very civil little flamewar.

Let’s review: before worrying about any software, we have to look at the humans. This principle spreads far beyond software. Want to design a park or a building? Look at the humans. This is why Frank Lloyd Wright was a fine artist and a lousy architect. Want everyone in the world to stop burning fossil fuels in cars? You had better know how you’re going to get them to switch to alternatives before you worry about the technical details of batteries or ethanol. Before writing a line of software, know whether it’s because you want a machine to churn away in a particular way (in which case, start coding), or whether you want people to churn way in particular ways around machines (which requires rather more thought about people than about software).

Here are some questions to ask yourself to get started:

What do we want the people to do?
“Be excellent to one another” is well and good, but between Christ, Buddha, Confucius, Zoroaster, St. Francis of Assisi, and countless others, there have been many social organizations proposed to bring this about, with notably little success. Unless you have a good idea of exactly what behaviors you want people to engage in, you’re not going to get very far. It wasn’t until we had a target rate of messages on our mailing list that we could try to guess what behaviors we needed from everyone.
Who are the people involved?
In the case study above, it was about twenty geeks scattered hither and yon in Switzerland. Many times on the Internet it’s “to whom it may concern,” since in the couple of billion people on the Internet, someone is probably concerned. It’s also worth asking how many of them are there, how much time do they have, and what background knowledge do they have?
How do I make the behavior technically feasible?
and, far more importantly, how do I make the behavior psychologically attractive? Answering the psychological side usually makes the technical side fairly clear cut.

The next time you start a project, first, look at the humans.1

1 I do not mean to be discriminatory here. If it is the behavior of gerbils or tentacled aliens that concerns you, looking at the humans is probably a bad idea, and I encourage you under such situations to look at the gerbils and/or tentacled aliens very closely.