RE: 3. "Change Reports - ProtocolOps or Primitives"

["Albert Langer" <Albert.Langer@xxxxxxxxxxxxxxxxxxxxx>]

Steve,

I've now caught up with all other responses but am still unsure what you are
getting at on this one. I'm omitting some and just focussing on the
bits I *think* I understand. But they make so little sense to me that
I'm still not sure I really do.

[Steve]
[...]
The problem is that a list of all the standard LDAP or X.500 update
operations does not describe all the possible update behaviours
exhibited by directory servers.

[...]
For the protocol design there was a discussion on whether the user updates
would be conveyed in the replication PDUs in their original form, which
was then broken down in to primitives by each replica to be processed, or
instead as the broken down set of primitives determined by the originating
server. It was a given at that time that the updates would be broken down
into primitives at some point, so it naturally came down to a choice
between LDAP update request or replication primitives in the protocol.

The replication primitives can describe exactly how a directory server
executed a user update request without regard for the access protocol,
protocol extensions, conformance level, support for operational attributes,
etc. The small set of replication primitives is sufficient to describe a
much larger set of current and future directory server behaviours, as
well as being the basis for reconciling updates.

[Albert]
Ok, I understand that URP took as a given that operations would be broken
down into primitives at some point, whereas MDCR took as given the opposite
principle of treating operations as a whole ("atomicity"). I still don't
see how anything follows from that as to the desirability of using the ASN
for protocolOps or primitives in the protocol "on the wire". As far as I can
see,
either approach could switch to either format for reports generated by
originator DSAs. That highlights the advantages of separating report
propagation from update processing.

You still *seem* to be raising an issue of primitives supporting non-LDAP
directories better than protocolOps. This WG is about LDAP directories.

There could well be other LDAP replication protocols, eg broadcast,
defined in future. They will still take as given that the directory supports
what standard LDAP clients expect it to support.

Update procedures (whether based on reconciliation or conflict
resolution) should be applied at each DSA independently
of propagation of originator reports to ALL DSAs. That maps well to the
fact that any LDAP client can update any LDAP server - the whole point of
LDAP.

A list of all the standard LDAP change operations
*does* describe ALL the possible change behaviours that LDUP can define
standards about. If some behaviour cannot be expressed in terms of standard
change operations then its replication cannot be standardized. They are
carefully
aligned with X.500 so that they can be mapped with X.500 DAP change
operations.

> On the other hand you might more plausibly have meant to say that the
> original protocolOp as received by the originating DSA may
> include such
> non-standardized information, or not be based on a DUA
> protocolOp at all,
> and therefore should or could not just be copied verbatim by MDCR.

[Steve]
That's what I was saying.

[...]

I think you might be underestimating how difficult it will be to translate
arbitrary extensions, controls and alternative access protocols into
canonical protocolOps. For one thing you will have to be prepared for
a single user update to map into multiple protocolOps affecting multiple
entries. I can think of a number of features in our directory and other
vendor's directories that would require this.

[Albert]
Whatever the difficulties, that is exactly what an implementor has to do to
generate either protocolOps or primitives at the originator DSA. Allowing
arbitrary extensions to affect the propagation of change reports would be
even more nightmarish. Some implementations will benefit from being able
to copy the DUA protocolOp verbatim. Others will not, but anything extra
they have to do is their problem, not ours. As long as they *can* map
their directory to LDAP standards, they can benefit from LDAP standards.

Whatever the "features" in your directory that have led you to contemplate
URP, I can only suggest that you fix them and that other vendors with
similar problems do likewise. The IETF standards process is not about fixing
vendor problems resulting from failure to comply with existing standards.

For example backlinks for pseudo referential integrity such as a memberOf
attribute in a User entry that lists what groups the user is a member of
could either be efficiently generated automatically at each compatible
DSA or explicitly generated at the originator so that they would also
update DSAs that have no such facility.

Only the former makes sense to me, for future standardization of backlinks,
and also for current interoperability, as the latter would still result
in inconsistent backlinks from changes originated at DSAs that do not
support that facility.

But either way, allowing DSAs *other* than the originator to generate
ANYTHING based on its arbitrary features while propagating, or allowing
originators to generate anything that cannot be expressed as standard
change operations, is a certain recipe for incompatability.

Checkout the ASN for an LDAP protocolOp in RFC 2251. The subset for change
operations includes ALL the information that is included in the set of URP
primitives generated at an originator DSA and replicated in
a framed operation to its neighbours, except for the entryIDs.

An MDCR "report" adds the entryIDs and therefore includes ALL the
information required to apply URP procedures.

Simply re-using the existing ASN just makes the MDCR protocol much
shorter to specify and easier to understand.

If there was any reason to do so, an MDCR report could be reformatted as
a sequence of URP primitives, adding the DN of the entry and the report and
version numbers etc omitted by URP. I still cannot see any reason to do so.

The critical distinction in report propagation is that MDCR replicates that
report unchanged to every DSA and each applies it independently, whereas URP
merges or reconciles reports while propagating them, thus adding to the
inherent problems of merging or reconciling them at all.

If for some reason, URP "reconciliation" was desirable, it could be done
independently at each DSA in applying the reports (or primitives) while
still having the same approach as MDCR in propagating every report (or
every originator primitive) to every DSA.

Complete separation of report propagation from update processing is a
more natural "layered" approach to protocol design and makes it much
easier to understand and predict the result, achieve convergence etc etc.

A possible explanation for whats going on here is that certain vendors may
have
adopted a "state based" approach to replication that cannot be reconciled
with
the atomicity required by LDAP standards, and are trying to get IETF
endorsement
of a revision of the standard through the back door.

If that IS what is going on, they should instead propose up front a
revision to LDAP v3 while it is still a "proposed standard" rather than a
"draft" standard or "standard".

A vendor dominated WG revising a base standard won't cut it at final call.

If a group of vendors want their own "industry" standard rather than either
X.500 or LDAP, they can easily establish a forum to define it. LDAP has been
adopted by vendors precisely because clients based on real standards
make more sense than trying to combine the features available in various
proprietary clients. Exactly the same will prove true when servers are
replicating to each other as well as interacting with clients. You either
standardize or you don't interoperate.

[Steve]
I regard efficiency on the wire as largely a non-issue for replication.
My experience with X.525/DISP is that supplier DSAs can pump
changes through to consumers at least an order of magnitude faster
than the consumers can process them. A super efficient LDUP protocol
won't make much overall difference.

[Albert]
I agree that the number of bytes is a non-issue. As a separate matter,
allowing replication out of sequence requires a restart from the beginning
to ensure update vectors are synchronized whenever a replication session
is interrupted. That is most likely to happen on connection oriented links
where
bandwidth is most costly.

I mentioned in email to Ed and will now repeat here, that this is especially
a problem for "sometimes connected" or dialup DSAs that may need to
interrupt
a "supplier push" session to a Hub from another dialup DSA in order to
obtain
a session for the same replicated area, given that only one consumer session
can be handled at a time by the Hub.

Likewise, use of timestamps requires a full reload rather than incremental
to get back into sync after failures, which causes bandwidth spikes.

Both of these "features" of the current architecture draft are not inherent
in URP but easily fixed. So far nothing has been done to fix them.

Unlike the number of bytes in the syntax, they would have a significant
impact on bandwidth requirements.

> I also believe it is important not to impede future extensions to the
> replication protocol to cover future standardization of
> additional features
> such as "signed operations". URP impedes this by nuking
> operations so that
> the very concept of an operation has been lost once it has
> been subjected to
> atomic fission and its constituent elements sent wandering through the
> replication network. MDCR preserves the concept of a "whole"
> operation (ie
> an "atomic" operation), for other reasons, but has the
> additional benefit of
> not impeding future developments such as signed operations.o

[Steve]
Signed user operations are no more an issue for LDUP than they are
for DISP (i.e. irrelevant). X.500 has signed operations and doesn't
use the original DAP operations in its replication protocol and I've
never found a problem with that. In any case, a canonical MDCR change
report would invalidate the operation signature as surely as breaking
the operation down into primitives.

[Albert]
X.500 DISP is single master and replicates operations, whatever the
format.

RFC 2649 (experimental) specifies a "signedAuditTrail" which includes
a set of changes, each of which has a sequence number and a signed
operation. A DSA can add its own signatures verifying the signed
operations and a DUA can check the signed operations in sequence to
verify that the entry state it is relying on is supported by signatures
it trusts.

This is particularly important for multi-master replication because
every DSA and connection is an independent point of vulnerability. In
that environment, certain critical functions are more likely to need
verification of the origin of the data as they cannot just trust a
SINGLE DSA and the path from it, but must trust an entire network of
them, not all under the same management.

URP cannot meet any such requirement because the entry state is NOT
the result of applying a sequence of operations, as required by the
LDAP/X.500 data model and assumed by people writing both applications
and extensions for it. RFC 2649 is another excellent example of
this assumption at work.

MDCR can support signed operations because entry state IS the result
of a carefully maintained sequence of operations, as required for
atomicity.

Perhaps the fundamental problem with non-atomicity is best illustrated
by that in itself - nobody could guarantee the integrity of the result
with a signature, because the result does not in fact have that integrity.

Actual implementation of that support would require liasion with LDAPEXT
to define replication of the "signatureIncluded" field of the
signedOperation
control and what to do when the originator DSA generates a signature on
behalf
of the DUA.

Rather than carrying both the DUA original protocolOp used for the signature
AND either a set of primitives or a canonical protocolOp used for report
propagation, it would be better, if possible, to avoid that duplication by
defining a canonical protocolOp to be used for signed operations. (Perhaps
using DER, rather than BER and mime as specified in RFC 2649).

This issue is "irrelevant" for URP since it cannot possibly support signed
operations anyway. That does not make it irrelevant for LDUP.

The fatal mistake of treating the LDAP v3 standard requiring atomicity as
irrelevant is what makes URP irrelevant for LDAP and therefore for LDUP.

Incidentally, liaison will also be necessary concerning the "zombieObject"
used in RFC 2649 to maintain an audit trail for deletions.

[ACI issues moved to item 9]