RE: Comments on ldup-replica-req-00.txt

[Alan Lloyd <Alan.Lloyd@xxxxxxxxxxxxxxxxxxxx>]

Thanks John for the reply. Comments follow.

> -----Original Message-----
> From:	John Merrells [SMTP:merrells@xxxxxxxxxxxx]
> Sent:	Tuesday, April 21, 1998 10:22 AM
> To:	Alan Lloyd
> Cc:	ietf-ldup@xxxxxxx; ldup-repl@xxxxxxxxxxxxxxxxxx
> Subject:	Re: Comments on ldup-replica-req-00.txt
> 
> Alan Lloyd wrote:
> 
> > > I'm not sure how you managed to derive this from the text.  I see
> > > nomention of a client maintaining replica consistency between
> > > two servers.
> >         Perhaps it was that LDAP - as a client-server ACCESS
> protocol
> > that swayed me. Is LDAP now a LDRP or something (replication) and
> server
> > to server operations.
> 
> Servers can behave as clients to each other.
> 
	Of course - its just that these servers just become "normal"
users of the directory that must be authenticated and if one changes the
credentials of a server in one server - how does that get into the
second - unless servers need to use their old credentials depending if
they have replicated them or not... is this chicken and egg stuff.

> >         Does this mean changes to the LDAP RFC?
> 
> Probably, in the form of new controls, and possibly extended
> operations.
> 
> >         You see replication needs to be initated by either server
> and
> > replicas transferred in either direction so its either a
> bi-directional
> > LDAP server to server with transaction control and server based
> > management and a lot of Gods help (this is not a religious statement
> > ;-))
> 
> It could be a session in each direction.
	With God in most cases it seems to be simplex - but I realise it
is bothways with LDAP :-) see above.

> >         OR its a third party client application controlling the
> > replication via LDAP.
> 
> I doubt anyone would want to implement that... unless it had someother
> synchronization functionality.
> 
	I doubt that too - its tends to lead to lots of problems - as
defined.

> >         I dont care  which - the transaction/atomicity problems are
> the
> > same. And the updates to the LDAP RFC may be needed.
> 
> I should think that new controls will suffice.
> 
	Thats a view as I would say it. Protocol fields  are easy to
write down . eg ip6 address is just 16 bytes - now try and deploy it and
make it work as a global network.
	Want to ask Oracle, Ingres or Sybase about transaction locking
systems and deadlock issues?


> >         Its seems that LDAP is getting more onerous with its DIT
> locking
> > and transactions and exposing this into the protocol. Is it likely
> that
> > X.500 DAP and DISP will be less protocol than LDAP and do more -
> more
> > effectively?
> 
> Depends what the LDAP replication architectural proposals are like.
	If they stay like DISP there is a chance of success - : -) ..
existance proof is good.

> >         But then how will I get the master entry if I need it. I may
> be
> > connected to a replica server for my address book details, but would
> > really like the military target information from the master in case
> its
> > been updated in the last ten minutes and has not been propagated
> yet.
> >         Users must have the mechansims to get to master info - in
> case
> > of inconsistencies caused by latent replica updates -
> >         See X.500 - Dont Use Copy, Copy Shall Do, its defined, it
> works
> > and its useful.
> 
> So far, no one's suggested this as a requirement.
> 
	May I suggest a feature which.........................

> I'd wonder though... that if your application has real-time
> constraints then
> you probably don't want to store your data in some loosly consistent
> store.
	X.500 permits inconsistencies - but leaves the ops people to
sort that out- but X.500 provides the services to always get the master
entry just in case the ops people dont make it.

> Also... if an entry is mastered by many servers... then you won't know
> which server holds the 'correct' value anyway.  So it doesn't sound
> like
> this buys you much.
> 
	In X.500 one does via the information being tagged as copies
(internally in the DSA because it arrived via DISP) - and with the
knowledge information in the replication agreements the replica DSA
knows of the master - and the via DSP (distribution again - an LDAP
deficiency) I can request of the replica I am connected to to go to the
master via DSP to get it by specifying Dont Use Copy - all seamlessly to
the user - you see replication and distribution in X.500 work together.
In LDAP land all the parts arn't there yet. So replication on its own is
very limited....


> >         How do we manage these tags (in attributes?) across tens of
> > replicas if not hundreds -
> 
> I'm worried that attribute value tagging might limit scalability, but
> amexploring a change log approach to see if a server can
> be limited to
> only knowing about the state of it's peer, rather than the entire
> directory server topology.
> 
> How does X.500 deal with this?
> 
	through replication agreements and using op attrs. 
	But the distinction must be made between the replication
protocol and the management of it.

	ie Utilities will be needed to reschedule protocol that could
not work at a specific time.
	Utilities will be needed to consolidate DIBs that might be
inconsistent, etc. 

	The replication scaling problem is reduced with distributed
directories in the first place.
	Replicated systems are defined for recovery/backup and or
geographically distributed loading. 
	There must be a strong desire to limit the number of masters and
replicas that have high update rates. If they do, then split them in a
distributed fashion under a distributed naming context. This just means
the area of replication becomes manageable.


	I think that the replication problems that LDAP will run into to
do with scale is a by-product of not having distribution. If LDAP deals
with this then one can connect servers together and just replicate
fractional parts of them to deal with the loading and the redundancy
issue. At the moment replication is being pushed to solve a User
connectivity issue - and that is unscaleable - ie all users entries in
all servers and thousands of replication transaction and synchronisation
issues to deal with.

	We took a very hard decision with DXserver in that we would
crack distribution and performance issues first (and that was a hard
one) because it means dealing with naming, knowledge, access controls,
domain based issues and distributed system concepts AND performance of
DSP interchange + search results correlation, etc.
	and then we dealt with replication. Many servers and DSAs have
not dealt with this distribution too well - so they blast "replication"
as the strong point ...But that does not scale ie. the more replication
there is - the worse it gets. Then one has to ask why and then one is
forced to deal with distribution.

	In closing - the replication work must define the replication
information model - eg are replicas marked, what is the processing
around agreements (re knowledge of master/copies) - what are the
protocol issues re atomicity of the transfer - and then the utilities
needed to make sure there is system robustness and management.
	as said - just dealing with protocol in a large sacle system
design can be a non event.

	regards alan 




> >         Oh really - how does the server know its a partial entry or
> not.
> 
> The replication agreements define the area of the tree to be
> replicated,including an entry and attribute filter.
> 
> >         See X.500 Copy shall do...
> 
> Eh? See X.511. DAP - controls  for acceptance of partial, replica
> entries in a search response.
> 
> 
> John
> 
> --
> John Merrells
> Netscape Communications
> Directory Server
> Software Engineer
> 
> http://people.netscape.com/merrells
>