Issues with Spark Properties not Fully Loaded from zingg.conf in spark-submit Job | Zingg Community

Zingg Community Icon

Zingg Community

Issues with Spark Properties not Fully Loaded from zingg.conf in spark-submit Job | Zingg Community

Samarth G.
·
Looks like you might want to try the --conf option
Samarth G.
·
Okay, doesn't look like that is the case
Rajesh P.
·
yes, --conf is specific to the config.json.env file that we are passing, which contains the field definitions stuff. --properties-file is for the zingg.conf file as mentioned here in Zingg docs https://blog.infostrux.com/identity-resolution-with-zingg-ai-snowflake-and-aws-emr-for-the-canadian-football-league-22cf0850ab53
Samarth G.
·
Yup, my bad.
Samarth G.
·
Btw Rajesh P., the blog you're referencing does use the zingg.sh executable, and not spark-submit directly. might want to recheck that
Samarth G.
·
Are you running it on AWS EMR?
Rajesh P.
·
Yes I am running on AWS EMR, and to the above note, i believe the arguments, like --conf, --properties-file would still hold true right while running the zingg job right ?
Samarth G.
·
Yeah, should be fine. the zingg executable just translates the --properties-file argument as is, but doesn't really have a --conf option I think
👍1
Sonal G.
·
See if there are emr specific ways of passing spark configuration values?
Rajesh P.
·
yeah Sonal G., EMR too takes the properties like spark-submit command... checked here https://repost.aws/knowledge-center/emr-set-spark-parameters
Sonal G.
·
Sorry i dont see the --properties-file option in the link you shared.
Rajesh P.
·
Sonal G., i have tried to pass the --conf option as well and passed the spark zingg.conf properties file. It is not able to run... But, when i tried to run using --properties-file by passing the zingg.conf file, it is able to pickup few spark properties ( like spark.executor.memory, spark.executor.cores ...) , but not all the spark properties (like spark.serializer ) mentioned in the zingg.conf file... (you can see in the screenshot above)
Sonal G.
·
from what I understand, the --conf option is only for limited properties, not a whole file. the link you shared has clear instructions on setting the default properties. have oyu tried that?
Rajesh P.
·
yes...
Sonal G.
·
there is nothing in Zingg code to selectively load spark configuration. I would recommend checking the EMR docs and support to see whats going wrong.