From the perspective of the prospective iOS developer, the relevant question is: is it possible to make a living in the iOS space?

Sturgeon’s Law: 90% of everything is crud.

The first fallacy in analysis of the iOS market is that people take the market gross and divide by the number of apps and say “The average app makes $1,634 a year!” (or something to that effect).  This is not meaningful for an app developer, because it both includes the “superstars” that are making millions in a day, and the whole of the long tail.

So more intelligent analysts tend to cut off the top 1% or so. Unfortunately, this just makes the analysis worse. However, there is an additional fact that presents a more reasonable picture, and that is that 90% of everything is crud.

Now if we assume that iOS consumers are providing a loosely-coupled assessment of the cruddiness of a piece of software when they make a purchase, the iOS market starts to look a lot more reasonable.

In the simplest case, where the frequency of purchase is linearly associated with the cruddiness of the software, we can discard the bottom 90% of the app-store purchases from consideration. After all, you’re not going to make cruddy software, right? Now the graph starts to look a little better. We’ve discarded the top 1% of the sales chart and the bottom 90% of the sales chart.

The graph that results is a much more linear graph, with the top app in this zone selling a few thousand a day and the bottom selling several hundred to a thousand a day.  You can jigger the numbers to suit your preferences- for example, you could decide that some significant percentage of people have poor taste or don’t recognize quality – but most of the variants I came up with still amount to a survivable to comfortable revenue stream for a 3-5 person company.

This analysis is relatively simplistic. More advanced assessments would be based on revenue-per-app as opposed to sales-per-app.

Of course, you still have to make an app that isn’t judged as “crud”.

The problem that made me re-implement was that I was polling for new posts, and if the user made a new post I wanted the wait between poll attempts to go back to the minimum.

The main improvements of this version over the enfranchised mind version are:

• Multiple active polls
• Ability to modify the ajax settings after the poll has started
• Can make a one-time change to the current polling interval.

You can download or contribute from the project page.

Usage:

//All poll options are optional.
var pollopts = {
  //A random name will be assigned if you don't assign one
  name: 'name-of-poll',
  //minimum wait between calls in msec
  min_wait: 6000, // default: 1000
  //maximum wait between calls in msec
  max_wait: 12000, // default: 30000
  // amount to multiply the wait by if the data is unchanged.
  wait_multiplier: 4, // default: 2
  // log poll events to console.log
  doLog: false, // default true
  // A function that overrides the default calculation
  // for how to change the wait if nothing has changed.
  // Should return an integer
  adjustWait: function(xhr, textStatus, current_wait) { ... }, .
};
// simplest way to start polling.
// ajaxopts are defined as per
// http://docs.jquery.com/Ajax/jQuery.ajax#toptions
var mypoll = $.poll(ajaxopts);
// start polling with custom poll options
var mypoll = $.poll(ajaxopts,pollopts);
// get a poll object, but don't start polling
var mypoll = $.poll(ajaxopts,pollopts,true);
// can also do $.poll(ajaxopts,null,true);
// stop the poll
mypoll.stop = true;
// make a one-time change to the poll wait interval
mypoll.setWait(500);
// change the ajax options for the live poll
mypoll.setAjaxOptions(ajaxopts);

Social identity

Social identity – the kind of identity that exists between people independent of any technology – is the bedrock of ‘who we are’ to each other.  Social identity is clearly an innate, evolved system; we can tell this because it is clear that social identity exists in other social animals, and is absent from asocial animals. Humans have a fairly elaborate system of identity that is strongly cultural, although it generally has significant biological components; gender, heredity, and genes (i.e. beauty) are some examples.

Another indicator of the “built-in” nature of our social ability is the fact that is constrained. Estimates vary, but most studies suggest that we can maintain a (self-constructed) awareness of the relationships between a few hundred other social identities, and even then the knowledge is contextual and thus we cannot be aware of the full set of relationships, but rather given a particular identity we can be aware of its relations to others.

There is quite a lot of scope for disagreement with this estimate, but unless this estimate is off by more than five orders of magnitude, it is impossible for an individual human to have a model of all the relationships between all humans presently on earth. It seems reasonable to suggest that even on the scale of most present human settlements, most individuals have no awareness of all the identities in their settlement. This suggests that a global system of identity would need to provide technological assistance to people in order to allow them to assess relationships at larger scales.

One thing that is consistent across all cultural notions of identity is is that they are not unique, persistent, or immutable. In a certain social context, a person may be the ‘life of the party’; in another the ‘diligent worker’, in yet another the ‘enemy’. Social identities are fabricated from within the context of social interaction, and as such they have a scope and persistence that is inextricably associated with the participants in and circumstances of any particular social process. A successful system of identity must take these characteristics into account.

Supersocial identity and the rise of the Individual

Human societies long ago outstripped the capacity of individual humans to hold them in their heads. As a result, the social organizations that formed around large groupings of people developed technological extensions of identity to allow them to work on a larger scale.

For thousands of years, it was sufficient for societies to reify social identity as distinct from digital identity. A simple encapsulation of this type of identity is the expression “The king is dead: Long live the king.”  Caste, class, and other ‘type’ identities were used by to reduce the number and complexity of social relationships that had to be considered when ‘innate’ social ability was insufficient. A peasant and lord may have never met before, but when they do their reified social identity provides them a framework of interaction that doesn’t depend on their knowledge of each other.

Despite these reified social identities, the ‘individual’ identity remained essentially the same as it had for the prior five million years; determined by actual interactions with specific other individuals. A detailed review of the emergence of “technologies of individuality” is beyond the scope of this essay, but for our purposes we can consider them to have begun with the census. The example familiar to english speakers is the Domesday Book, which contains the results of a survey commissioned by William the Conqueror to figure out exactly what he had conquered. In England, at least, this was the first time that many people acquired a second (last) name, so that they could be differentiated from others in the survey. These surveys eventually led to systematic registries of individuals, independent of their participation in significant events (birth, death, marriage, etc.) Only then did the average person begin to acquire an “identity” at the superorganization level.

These individual registries became the bedrock of greater participation of individuals in the supersociety; with these identities in society, individuals rather than families or other associations could own property, have formal relationships through written contracts, and the like. Ultimately they established a supersocial notion of a human as possessing a unique, persistent, and immutable identity that is the same in all contexts.

At the same time, these systems of individualization were applied to reified social identities. The primary example of this is corporate charters; in the beginning, someone with strong type identity – usually the king or queen – created an individual identity and granted it some of her characteristics. In english, at least, the basic words we use illuminate this origin: an organization is ‘incorporated’, that is, given a body and made real. Today, corporations have a supersocial identity that is stronger than that of actual humans.

The pre-internet supersocial identity helped humans cooperate on scales heretofore unseen, and simultaneously created a new class of individual – the corporation. It also enabled dehumanization on a scale never before seen, finding its nadir in mass war and systematic genocide.

Digital identity

The digital world has largely failed to provide a substantial innovation on either social identity or supersocial identity. For the most part, distinguished strings are the extent of identity. An email address has served as the defacto ‘unique identifier’ for internet users – a (username)@(domainname) tuple – and some more advanced internet users have associated themselves with a domain under their control. The mass proliferation of email accounts and the rise of popularity of the social web has pushed identity closer towards being associated with usernames at particular sites (e.g. facebook) and the data associated with that name.

A reason for the success of this relatively simplistic approach is that it maintains the properties of social identity: its meaning is constructed by those who interact with it, and it may or may not have associations with other digital, social, and supersocial identities.

These identities in some ways are more liberating than standard social identities, because the individuals or groups creating them have much more control over what they elect to disclose about their identity, and the digital identity need not be associated with the supersocial individual.  However, much like  primary social interaction contains many ‘involuntary tells’ – expression, clothing, accent, etc – that allow participants to glean information about each other even if the other does not wish to share it, the name/data identities tend to have digital ‘involuntary tells’ that allow interested third parties to make inferences about the identity. These tells include unencrypted communications, visible ip addresses, domain name registrations, access patterns, and the like.

Just like the social world, the relationships between an individual’s identities can be explicit (use of your legal name on a website, explicit links to your accounts on other sites) or implicit (google for a username).  However, in social life there is always the explicit association of your body with particular identities, which serves as the ultimate, or primary association between an individual and their identity. Even modern supersocial individualization depends on biometric data, though corporations are more free from this constraint than people.

The association of identity and body is is hard to fake in the real world – disguises and plastic surgery are some examples of attempts to do so. The primary method for divorcing body and identity, throughout time, has been to simply re-associate one’s body with a new collection of people, so that a new identity may form. In contrast, name/data identities are extremely easy to disturb in ways that are more difficult for social identities.

One can create accounts on websites with the same username in use elsewhere on the internet, deliberately or involuntarily, and thus confuse the identity for any third party. This confuses the association of identity with username, and the methods for disambiguating them are more challenging than those for confused identity in the real world. Usurpation of identity is also easier in the digital world than the real world; if someone’s credentials are hacked, it is extremely difficult to differentiate the compromised identity from the ‘real’ one.

So the increased social freedom of constructible identity comes, in the current implementation, with the increased vulnerability to usurpation and pollution. A better system of identity should prevent allow disambiguation, and provide methods at least as robust as those present in human society for detecting forgery and usurpation.

The way forward

Cryptographic keys have existed as a stronger form of identity for some time. The limited computational capacities of the early internet and the peculiar ideology of the Cold War kept cryptography out of the early implementation of the internet, which severely constrained the ability of technology to aid identity. For a variety of social reasons beyond the scope of this essay, when cryptography did start to be used widely on the internet it was highly centralized and used primarily for securing specific communications.

Cryptography can help identity in several ways:

• It can be used to ensure that you are engaging in communication with the identity you mean to communicate, and not another.
• Digital artifacts can be proven to have been signed by a particular identity.
• Identities can be proven to be distinct.

The common internet user only encounters cryptographic identities during secure web use, where an asymmetrical cipher is used to allow the communication to be secure. The keys used in this system are generated via a hierarchical system, where a small number of ‘absolutely trusted’ keys are used to sign, or validate, keys used elsewhere. Despite efforts to make these keys serve as a form of identity, the fact that the absolute trust pool is maintained by technology implementers means that the identity granted has never taken on much social or supersocial relevance.

Early adopters

The people that develop the internet were the first to require a stronger form of identity, and sufficiently motivated to implement one and participate in sometimes-difficult rituals to maintain them. In the open-source world, this has taken the form of cryptographic keys, generally associated with tokens relevant to supersocial identity (i.e. real name) as well as some ‘unique’ string (email address). Because no central signing authority is trusted, individuals sign each other’s key to indicate validation of their identity. The social practice is to engage in these key-signing activities only during an in-person meeting. In this way, the identity of the key is associated with the corporeal identity of the user.

This practice works well, and has allowed distributed, non-hierarchical cooperative behaviour on a scale previously unseen. Its current limitations are its requirements for a moderate to high degree of technical skill, and the lack of internet-scale standards and implementations for perusing the webs of trust the users have generated.

A graduated approach

A way forward would be to associate keys with all actors on the internet, be they software, organizations, corporations or people. Since “having a key” is a necessary prerequisite to any more advanced identity use, the keys must be generated on-demand, and with no validation requirements.

So everyone would have keys, everyone could sign things, and everyone could encrypt things.

The in-person key-signing of early adopters has an important purpose, however: it allows you to definitely associate an identity token – the key – with an actual identity – a person you met at a gathering.  By inference, it allows others to also infer identity, even for keys operated by people they have never met, because they know the identity of one or more signatories of a given key.

Therefore the replacement system must have a way of taking these profligately distributed keys and associating them more strongly with an identity.

Automated key association

There are some automated mechanisms for strengthening the identity associated with a key. Some of them significantly impair the multiplicitous, ephemeral, and mutable properties of normal social identity, and so have limited application to keys we wish to associate with human identities.

The primary mechanism for identity strengthening would be a globally accessible system for recording the activities associated with a key. That is, when a key is used for a communication tranasaction, or signs a file, that event could be recorded. The set of events associated with a particular key provide an identity that is substantially more substantive, and in many cases quite sufficient.  Care must be taken when deciding which properties of events should be included in the record, as excessive detail would make it easier to determine ways to usurp or subvert the keys.  For example, this suggests that transport layer details (ip address, mac address, OS, software version, etc) should not be included in any record.

For software identities – keys used to identify a particular instance of running software – recording how well particular keys conformed to accepted interactions would provide sufficient information to determine whether a given piece of software ‘should’ be trusted to perform specific actions.

Generating identity through social interaction

For actual humans, the situation is more complex. Meeting the requirements of disambiguation and usurpation prevention while simultaneously preserving the ability for social identities to be mutable, ephemeral, and multiplicitous is an interesting challenge. The early adopters of key-based digital identity have not tried to address these use cases at all. In particular, the focus has been on identifying an original uniquely, and while there exists support for changing keys, this support exists to maintain an immutable identity in the face of key loss or compromise.

Therefore, a new approach is required. If we consider what remains of identity when we strip it of type – class, race, creed, etc – it seems what remains is a pattern of interactions between people.  The simplest form of identity interaction is generally a declaration of an identifier; “Hello, my name is Ethan. What’s yours?”. The voluntary association of a social identifier (name) with the physical identifier of one’s body is the beginning of creating a social identity within a group.

Attestations – written, signed, and countersigned, then stored in a public repository – are widely considered to be ways to formally assert statements about oneself. One could argue that all contracts – though they are usually not exclusively descriptive of the signers – contain some level of identity assertion.

Therefore, an improved system could differentiate the quality of a human identity by specifying particular declarations to be made using a key. A structured textual statement that certain pieces of information (a name, a hobby, whatever) are associated with this key would be the most basic form of declaration. An audio recording of the key’s owner, declaring that they are the owner of the key, could be another. A video could be a third.

All of these attestations create the risk of involuntarily associating an individual with a key immutably. Therefore the system must provide a way for these attestations to be made and verified without allowing global access to the attestations.

This system would also support, at each level, signing of these attestations by others. This would provide a similar purpose to the in-person key signings, but would allow the development of the web of trust independent of the capacity of the participants to meet physically.

Associating identity with social groups.

Social groups in human societies are fluid; they form without notice or even, perhaps, the awareness of their participants. There are also relatively rigid, long lasting social groupings. Any better identity system should support these disparate groups.

A way to address fluid social groupings is to treat any set of mutually co-signing identities as having an identity, and thus generate an appropriate key for that group. All members would be given the ability to act using that key.

Since this implies creating new keys any time set membership changes (or at least every time set membership decreases), this would not address the need for persistent social groups who retain identity in the face of membership changes.

This, like the ‘level’ of identity outlined for individuals above, should be a graduated thing. The first step would be to name a given set-key; this would allow its identity to persist in the face of key changes.

For whatever reason, many existing social organizations choose to limit the number of members who are allowed to act as the organization, and further to constrain the extent, or the area, in which a member may act as the organization. To support this kind of organization, the system either must allow all-party keys to be ‘downgraded’ to a key where only a subset of the members have access to the private key, or it must allow the creation of such keys ‘de novo’. A later post will discuss ways to enable this feature.

Since group keys are identical to individual keys in functionality, ‘specialized keys’ for an organization may be generated simply by associating different groups with each other – that is, organizational keys can sign each other to cause a superorganization to exist, and optionally name them.

Further steps

It seems clear that public key cryptography offers the technological basis for identity. By automatically and dynamically creating individual and group identifiers we support the fluidity of social identity, and through signed attestations and keys we can create webs of trust.

In later posts we will discuss the implementation details of the identity system outlined above.

• it’s heavy – like 50mb!
• writing for it is not like writing a web app, though it has similarities.

Basically I think it won’t help us much with the “degraded” scenario – where someone has “only” a browser to interact with the system – and that’s kind of a non-starter.

So I started to think about what we *really* need on the local system.

Basically the only issue with being just a web app is that we need access to the local version control install. For the most part, this could be addressed by having “whatever we install locally” be itself a server. That seems like a good strategy, because then the remote/local difference is pretty arbitrary.

The user experience issue would be how we interact with external editors.  The only way around the security restrictions I see is for the local server to actually do the open-file-in-editor, and the web app just issues said instruction to the local app.

There are a bunch of value-adds to this – the “server” could be any of these “vcs in a bottle” systems that is nearby, while the “client” could be your phone or whatever.

So I think this is the approach we’ll take.

I’m now investigating mercurial instead of git (windows/mac installers are available) because it plays nicer with the use-cases I’m interested in supporting and it has more of an “API” – plus it’s in python so I can grok it.

• XML/XHTML differencing. DaisyDiff seems to address this problem. XML dialects rule structured documents these days, but line-comparison gives you bad results and often malformed documents. This good survey of XML differencing got me to DaisyDiff, and it links to other relevant issues.
• Image differencing. Perceptual Diff compares raster images. Image diff is a nicely simple php image differencer. I can find nothing on higher-level image raster differencing ( comparing layers and whatnot ) and nothing at all about comparing vector images (though SVG might be amenable to xml differencing).
• Video differencing. Not much here. Some papers, no open-source software I can find (not a surprise, as most of what I can find is related to digital restriction management and copyright enforcement). Current takeaway is if the video collection isn’t in the same format, you’ve not a prayer.
  • A paper on scene boundary detection presents some algorithmic method for finding scene ends in a video, which could be used as a starting point.
  • More papers I haven’t read through yet:
    • Identifying distinctive subsequences in multivariate time series by clustering
    • Clip-based similarity measure for hierarchical video retrieval
• PDF differencing. Adobe offers pdf differencing. Not clear if it can readily be used in a server environment. Other tools I could find were windows only, and not geared towards generating the differences as a product itself.
• “Office document” differencing. There are some scripts for open-office doc differencing. I suspect that using a good xml differ would be sufficient.

General differencing

• diff-match-patch is google’s differ.
• Smart Differencer attempts to compare semantic differences between code at a “language construct” level.

Current Conclusion

• Biggest takeaway: constrain differencing to supported formats. I’m pretty close to convinced that we should not offer the ability to manage documents in unsupported formats at all, given how much the user experience of version control degrades when no differencing ability is present. Transcoding all input into supported formats for purpose of differencing is another option.
• Most of the rich document comparison that chit (can has it together) should do can be done via xml. The devil, as it were, is in making meaningful user interfaces to the difference files thus generated.
• PDF differencing can be bought, but no good OSS solution appears to exist. UI issues would still apply.
• Video differencing is a rough field. Probably possible to buy algorithm engines for a big chunk o’ change. UI an issue that I haven’t seen addressed anywhere.

• Gitit – wiki using git/darcs as backend. Written in haskell, and super not for mortals (still have to use CLI)
• Theoretical discussion of using git for cms versioning
• Is git more than a version control system? – doing exactly kind of what I’ve been thinking of, using git / spidermonkey /bash to make a couchdb-alike

Anyway, either nobody really has anything production ready or it’s swamped by the number of CMS/document management systems that use Git for their source code management.

There are many use cases for this, but I’ll enumerate them, starting from ‘mostly connected’ to ‘mostly disconnected’:

• Server/peer availability – in a distributed system, many computers will be participating in server roles. They won’t all have high availability, so being “offline” from one or more peer as you seek a new one needs to be not disruptive.
• Transient connectivity quality – Even a highly connected individual experiences moments where their bandwidth is limited and/or expensive, and so ‘offline’ behaviour in those circumstances is helpful.
• Intermittent outages – Many people do not have an ‘always on’ connection through a cell modem or something similar. These people frequently have decent access, but there are valuable periods where they are completely without bandwidth.
• Frequent/predominant outages – There are many environments where transient connections are the norm. In these cases, offline use is predominant.

In all cases, what has to be addressed is not only the ‘simple’ technical requirement of maintaining a responsive UI and preserving state locally, but also the question of synchronizing the data when connectivity is restored. That issue is for a later post.

Offline options:

• HTML5 offers a standards-based approach for offline usage that is supported by the latest browsers IE 8(!).
• Google Gears offers offline storage as a plugin for older browsers, and it is built into Chrome.
• The Pre, iPhone, and Android support the offline features of HTML5.

The data models provided

• Web Storage allows for the preservation of key/value data, and offers ‘real’ session support. Keys are strings, and “Values can be any data type supported by the structured clone algorithm.”
• A SQL storage . Basically in-browser SQLite.

Initial Use

The main thing we want to use it for to begin with is storing cryptographic keys for the user.

However, at long last, XulRunner is on the 3.x firefox codebase, so it seems like this would be a great target for the application. It can be built for some smartphones.

So that would suggest that either we could get XulRunner to compile on iPhone/Android, or we could get the extra components we need (likely git, CouchDb, GnuPG) running on those platforms separately.

To achieve this in the browser, we must have a way of performing standard cryptographic techniques on data. Most of the existing cryptographic support in browsers is built around the Public Key Infrastructure, which basically boils down to “trust the corps that have root certificates”. Furthermore, it is entirely concerned with stream based encryption, and doesn’t help content-encryption.

• JavaScrypt – an implementation of AES in javascript.
• Enigform and FireGPG provide a firefox exension that allows GPG-based encryption of http content, as well as digital signatures, etc. This is essentially the feature requirement we’re looking for, but Firefox/Moz only.

For the moment, I’ll mostly be writing about technologies and issues that are germane to our implementation; it will probably be about a year before there’s much for the end user (sorry).