I am trying to read a csv file in cloud datafusion. The csv file uses a multi-character (i.e. ~^~)delimiter. When i try to parse the column using a custom delimiter the tool only considers the first character and splits the file accordingly. I end up with more columns than what is required, plus since the data may have the character even the split is not perfect.
I tries using several patterns not just the one mentioned above, but each time its the same result
How do i parse files where the data is delimited using a multi-char delimiter. Is there a setting that can be used. Is there a separate transform that can be used to parse the file before using the wrangler?
In the assignment i am working on i an limited to use only cloud data fusion as my etl tool