Can Zingg De-Duplicate Data Without Using Spark?

Hi What are your requirements?

Community version of Zingg is a Spark application. However, Zingg’s docker abstracts away all Spark/Python/Java dependencies and you can run it easily for few million records. You can explore managed Spark provided by Databricks, Fabric, AWS etc as well. On the Enterprise side, we also have a Snowflake Container app which runs without Spark.

HTH

Hi Sonal G., what kind of latency can we expect in an incremental setting with the enterprise version?

Depends entirely on the incremental load, existing size of identity graph and cluster settings. Typically a few minutes.

If the incremental “batch” is a single event, does it still take that long?

few seconds then - mostly spark job setup and shutdown will the major factor. whats your need here? would be interesting to understand and see whats the best path forward

We’re looking for realtime dup-detection on a single event. So ideally sub-second.

If most of the latency is from spark, I’m wondering if it’s possible to run the incremental single-event classification task without spark.

Not without code changes, as the feature engineering and prediction are spark jobs right now.