because ideally I wouldnt' want it to load all the data every time.

Sounds right Brandon M.! We have built the enterprise version keeping this exact problem in mind. Zingg Enterprise assigns persistent globally unique identifiers and handles incremental loads with cluster evolution. Happy chat further if this is of interest.