1

I have an ETL flow through talend and there:

  1. Read the zipped files from a remote server with a job.
  2. Take this files unzipes them and parse them into HDFS with a job. Inside the job exists a schema check so if something is not

My problem is that TAC server stopes the execution because of this error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.talend.fileprocess.TOSDelimitedReader$ColumnBuffer4Joiner.saveCharInJoiner(TOSDelimitedReader.java:503) at org.talend.fileprocess.TOSDelimitedReader.joinAndRead(TOSDelimitedReader.java:261) at org.talend.fileprocess.TOSDelimitedReader.readRecord_SplitField(TOSDelimitedReader.java:148) at org.talend.fileprocess.TOSDelimitedReader.readRecord(TOSDelimitedReader.java:125) ....

Is there any option to avoid and handle this error automatically? There are only few files which cause this error but I want to find a solution for further similar situation.

4

2 回答 2

3

In the TAC Job Conductor, for a selected job, you can add JVM parameters.

enter image description here

Add the -Xmx parameter to specify the maximum heap size. The default value depends on various factors like the JVM release/vendor, the actual memory of the machine, etc... In your situation, the java.lang.OutOfMemoryError: Java heap space reveals that the default value is not enough for this job so you need to override it.

For example, specify -Xmx2048m for 2048Mb or 2gb

于 2021-02-24T20:41:57.277 回答
1

@DrGenius Talend has java based environment and some default jvm heap is awarded during initialization, as in for any java program. Default for Talend - Min:256MB (xms) & max:1024MB.. As per your job requirement, you can set the range of min/max jvm like min of 512 mb & max 8gb..

This can be modified from Job run tab - advance setting.. Even this can be parameterized and can be overwritten using variables set in env. Exact value can be seen from job build -> _run.sh .. But be careful not to set high as too high so that other jobs running on same server is depleted of memory.

More details on heap error & how to debug issue: https://dzone.com/articles/java-out-of-memory-heap-analysis

于 2021-03-13T11:23:21.353 回答