1

I have a java-spring-hibernate web application which call a my sql stored procedure which call user specific "event" information.

Requirement is to write the result set into a text file and send it to one ftp server .I have an allocated space of 4gb as heap size.I cant allocate more heap size than this. But heap size issue is coming when stored proc return large result set and processing it from server side

Current code :

1:Stored proc return large result set( say some million row)

2:mapping it to a list

3:write each row into a text file .For a single user write only 3 events .If he has more than 3 events ignore it(Please notice I am getting email based sorted result set from mysql). This process I need to do in server side

4:heap size issue came when result set size is large

Pseudo code:

 Stored proc
    delete temp table if exist
     get public event  into temp1 table based on condition
     get non public event into temp2 table
     create new temp tabletemp 3 by joining temp1 and temp2 and join with user table with some condition and sort it
    return select * from temp3 sorted based on email

Even after setting all required index it will take normally 20 minutes to finish Server side code

   1:Create File writer object
    2:call stored proc
    import javax.persistence.EntityManager;
    ...
    ..
    Query query = entityManager.createNativeQuery("CALL procName(param)");
    ....
    list = query.getResultList();
    ....
    for (final Object[] objects : evntList) {
                    final EvntDTO dto = new ExactTargetDailyMailDTO(
                            String.valueOf(objects[0]),
                            String.valueOf(objects[1]),....,....,);
    if dto.getEvntNAme() matches some condition(){
    //change event name and other params accordingly
    }
    eventList.add(dto);
    }
    ...
    for (int i = 0; i < eventList.size(); i++) {
    Dto dto = eventList.get(i);
    count=1;
    ...
    fileContent=dto.getEmail()+appendmore user info +dto.getEventInfo();

    while (i + 1 < eventList.size()
                            && dto.getEmail().equalsIgnoreCase(remindersList.get(i + 1).getEmail())) {
                    if (count < 3) {
                    ....
                   fileContent=fileContent+add next EventInfo 

                   }
    ...
    ..
                   bw.write(fileContent);
                    bw.newLine();

    }
bw.close();
fw.close();

Can anyone suggest me a better plan to do this?

Since I want to do some more operation in this list(like If he has more than 3 events ignore it),I cant think of taking 1000 or 100 000 data at a time and write it into file and repeat the process

Can anyone suggest me a better way to architect and do it?

4

2 回答 2

2

跳过第 2 步：将其存储到列表中。

这样，您就可以将所有流式传输的ResultSet 强制到内存中。

如果您在 sql 之前打开 File，然后为每个 rs.next () 写一行，那么您一次只能在内存中保存一些行。

如果删除后超过 3 个事件跳过。

或者

重写你的 sql 只返回 3 行（使用限制 3） - 这样就没有大的结果集来处理

关于您的代码的一些想法

这是您将对象接收到列表中的部分：

for (final Object[] objects : evntList) {
                final EvntDTO dto = new ExactTargetDailyMailDTO(
                        String.valueOf(objects[0]),
                        String.valueOf(objects[1]),....,....,);

  if(dto.getEvntName() matches some condition(){
  //change event name and other params accordingly
  }
  eventList.add(dto);

稍后您按顺序遍历该列表中的所有事件：

for (int i = 0; i < eventList.size(); i++) {

我不明白为什么不能做你在for循环中做的事情，而不是将它添加到列表中。

检查下一个条目是否与上一个条目具有相同电子邮件的部分- 您应该能够将其反转为查看当前项目是否与前一个条目具有相同的值。这样您就不必在列表中“窥视”。

您似乎也在根据这些电子邮件地址进行一些检查。如果您事先知道提醒列表中的值，您应该能够将其用作查询 ( ) 的条件AND email LIKE (?, ?, ?...以减少返回的行数？

于 2015-12-19T10:05:53.823 回答
0

这里的问题不是结果集，这里是你的代码。只需摆脱列表，并在结果检索循环中进行条件处理。并且去掉你添加到文件内容的部分，每次都直接写到文件中，而不是推迟。

于 2015-12-19T22:45:56.203 回答