Inscrit le: 10 Oct 2017
|Posté le: Ven 10 Nov - 16:17 (2017) Sujet du message: spark xml parsing with spark streaming
I am trying to parse an xml files using spark xml databricks package. I am using spark 2.2.0 and i do spark streaming to get the path of the XML file. Once i get the xml file i just get the sparksession.sqlcontext() and use read the xml file as follows , Its taking more than 30-45 mins to load the xml, This xml is about 5-10 MB max. And weird thing is that if i don't do spark streaming and just run this code on simple java program with sparksession. It works in minutes. May i know what am i missing?
I didn't find the right solution from the Internet.
motion graphics animations