Re: Why have a multipart document address?

[Jeff Rush <jeff@xxxxxxxxxx>]

On Fri, 2005-02-11 at 17:35 -0800, roger gregory wrote:
> Right as far as you go, we spent a lot of time on designing this in
> 1979, and never got as far as implementing it.

Ah, that there were notes of the paths taken and the dead ends
identified so I don't repeat them.

> Lets abstract a layer and remember that the documents don't have to be
> in their home server, they may have been moved somewhere else.  So each
> machine has a map of what spans of the docuverse it has, and has caches
> of, and what spans are cashed elsewhere as well as pointers as to where
> more information can be found as to the whereabouts of the info.

This sounds a lot like the thoughts leading to The Ent data structure.


> So a query about versions of a remote document can be answered by
> consulting the map of who knows where stuff about address A, and sending
> a query about A in that direction.  So when versions or links are made,
> the information about them has to be sent in the direction of the nodes
> which have to know about it.  We envisioned something like an enfilade
> that has a reflection of what is in various nodes but not all the
> information.  The point here is to have a distributed tree rather than a
> point to point crosspoint switch.  So that we get nlogn rather nsquared
> traffic.

This is a lot like several of the current peer-to-peer systems.  I've
been studying them for ideas on how to set up a distributed, anonymous,
dynamically-connected set of nodes.  The original Xanadu would have all
nodes be part of the Xanadu Operating Company, but the days of closed
data islands like The Source, the original Compuserve and even Prodigy
are long gone.

But I think tumbler maps have a lot of overlooked applicability to peer-
to-peer systems today.  And I use the best ideas I can find. ;-)


> How close are you to implementing this? and what base is it going on? 
> It's good to have thought this out a few steps in advance of that you
> are implementing, so that you know what direction you're headed. The
> single layer kind of system you describe seems to reflect the
> interconnectivity we currently have in the internet, and may be more
> appropriate, than the quasi hierarchical model we had.  Then again since
> nodes vary in bandwidth a lot more than storage size, maybe that should
> inform how this plays out.

I have a blob storage system that uses SHA-256 hashes for immutable,
non-coordinated unique identifiers, and the HTTP and Twisted Perspective
Broker protocol for access.  It runs as a standalone server.

On top of that, I have a Python object persistence system.  The Twisted
Perspective Broker API gives me much of the capability of the E language
re remote calls.  This of course runs as a client to the storage server
but is itself a server to a browser.

On top of that I have fully-functional tumblers and type-1 enfilades,
for hanging document blobs along the tumbler line and allocating unique
tumbler IDs for nodes, users, etc.

I'm still working on a satisfactory type-2 enfilade for handling links.
And I've got several prototypes for how to manipulate links in the
client, striving to find an easy, pythonic representation for
programming.

I've got nothing on user access control, in that I'm struggling with how
to handle distributed, pseudonomous identities, and what kind of ACLs to
use.  How to grant and revoke access, how to grant access to a subgroup
of people, those are still dark areas.  If all documents were to be
public, I wouldn't have this problem but I don't think that is practical
today.

I've got nothing on a viable ecurrency and paying people for their
information.  I feel something is needed but without a centralized,
trusted Xanadu Operating Company, we need a strong distributed
microcurrency system.  But far better people than I have worked on that
and failed.

I've got a Firefox browser extension, complete with Xanadu icons (bad of
me re copyright, I know but they look cool).  I haven't hung any
JavaScript code underneath it yet, but the Twisted Perspective Broker
provides something called Live Link, that let's JavaScript within a
browser seamlessly call to and from Python in the server.  I intend to
use that to write the front-end, as Firefox is about as universal a
front-end as we're going to get.

And I'm just starting to sketch out a backend <-> backend architecture,
hence my questions.  I want to add other servers as soon as possible.
When I created the Fidonet BBS system called Echomail, a newsgroup-like
email replication system, years ago, getting others to try it was
critical to rapid adoption and led to explosive growth.

I continue to advance toward a public system, occasionally dropping back
to experiment with the Green C code to extract another nugget of
knowledge.  I'm probably retracing many of the steps of the Xanadu team,
including some of the wrong turns.  I welcome discussion with others and
enlightenment from the Xanadu team but I'll keep plodding along
regardless.  Nothing I do is secret in any way and my code is under the
GPL.

I also formed the xanadu.meetup.com to try to drum up local interest
here in Dallas, but no one shows up for meetings.  And I've presented on
the architecture of Xanadu 2-3 times, but to very small crowds.  Few
remember Xanadu anymore and fewer still see why anyone would want it...

-Jeff