Hierarchy - Part 1 - Why we care so much about it

[Yaron Goland <yarong@xxxxxxxxxxxxx>]

Reading and writing data is expensive. Sure, if someone is showing up for the first time and wants to download a complete address book or a complete list of items for sale that's fine. We are here to help. But we find in most cases that our partners do a download on the first access and then targeted accesses from then on out. This means that in most cases they aren't interested in downloading everything. They just want to download targeted parts of the data. We like that a lot because the more scoped their reads are the fewer of our tables we have to do look ups in and the lower our expenses.

Similarly with writes, outside of creating a new item, it is all but unheard of for someone to want to update all the data in a contact or a "for sale" record or a structured data item. What is our most common case is that the partner just wants to update a handful of specific fields and leave the rest of the data alone. We would much rather just receive the specific fields to be updated because this means we only have to reach out to the specific tables that the data to be updated resides in. For example, a single contact's data can easily be stored in four or five different tables. If someone updates an entire record, even though they only want to change a few fields, we have to touch all the tables. If they just send us the fields to be updated we can touch fewer tables and, again, save money.

The flip side of the above issues is that if someone does want/need a big chunk of data to read/write we want to provide it for them in one call. This saves us bandwidth (which is a huge expense) and it uses our resources significantly more efficiently. E.g. if someone is going to ask for data items A, B and C it's cheaper (in general) for us to return all data items at once than to have to run three separate requests through our system to retrieve each item separately.

The same applies to writes. If someone wants to update items A, B and C we would rather receive a single update request that we can process quickly in the back end than receive three different requests that need to waste a lot of bandwidth and processing time just on protocol overhead.

The way we currently approach this scoping problem is via hierarchy. We organize our data hierarchically and we allow clients to address absolutely any level they want. That way they can download or edit just the scope they are interested in.

This is why we are so focused on hierarchy and its why having the ability to address any part of the hierarchy at will is so important to us. This requirement motivates the design you will see in companion to this e-mail.

        Thanks,

                Yaron