I am trying to accomplish a simple task of:
- Read a text file that contains SQL queries separated by ";".
- Perform each query in a separate transaction
- Be able to restart/resume the execution from the first query that failed.
I tried to do this with spring-batch.(version 2.1.8.RELEASE) The problems I am facing are as follows:
I failed to configure a simple out-of-the-box ItemReader
that would read several lines up to ";" and aggregate them before passing to the ItemWriter
.
I do not need a FieldSetMapper
I simply do not have no fields. My input is a text file containing SQL queries. Each query can take 1 or more lines and the queries are separated by a semicolon. Seems like it is impossible to define an LineMapper
inside a ItemReader
without specifying a FieldSetMapper
to a complex bean (I tried to set a BeanWrapperFieldSetMapper
to map to java.lang.String
but failed due the exception described below).
Question 1: why do I need a FieldSetMapper
with the prototypeBeanName
set to a complex object if all I want is to append all the lines in a single String
before passing it to the ItemWriter
? When I configure the fieldSetMapper
's property prototypeBeanName
to refer to the java.lang.String
I get an Exception saying that the number of names is not equal to the number of values. I debugged the spring-batch code and found that there are 2 values mapped to a single name: name=%SOME_NAME% values={%MY_SQL_QUERY%, ";"} and the Exception is thrown by AbstractLineTokenizer.tokenize()
line 123
Question 2: Can the line aggregation be achieved by using out-of-the box spring-batch ItemReaders
if yes, how they need to be configured? Do I miss something in my configuration? (see below)
configuration:
<bean id="basicParamsIncrementer" class="com.linking.core.exec.control.BasicJobParamsIncrementer" />
<batch:job id="rwBatchJob" restartable="true" incrementer="basicParamsIncrementer">
<batch:step id="processBatchStep">
<batch:tasklet transaction-manager="transactionManager" >
<batch:chunk reader="batchFileItemReader" writer="sqlBatchFileWriter"
commit-interval="1" />
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="batchFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource"
value="classpath:/bat.sql.txt" />
<property name="recordSeparatorPolicy" ref="separatorPolicy" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="sqlTokenizer" />
<property name="fieldSetMapper" ref="sqlFieldSetMapper" />
</bean>
</property>
</bean>
<bean id="separatorPolicy" class="com.linking.core.SemiColonRecordSeparatorPolicy" />
<bean id="sqlTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
<property name="names" value="sql,eol"/>
</bean>
<bean id="sqlFieldSetMapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="sql" />
</bean>
<bean id="sql" class="com.linking.core.model.SQL" scope="prototype" />
com.linking.core.model.SQL
is a simple class having 2 members of type String
: sql (for the query) and eol
(for the ';') It is used to overcome the situation that the AbstractItemReader
returns all the lines up to the separator (as a String
) and also the separator (in my case ";") as values
.
Question 3: In case the job that runs a bulk of SQL queries using just a single step reader-writer approach fails in the middle how can I resume/restart the job from the 1st failed query? Should I do it programatically by analyzing last run or can it be achieved by context configuration of the job-step and the related out-of-the-box spring-batch beans?