4

I was successful in reading a file while using multi-process environment using file locking and in case of multithreaded(singleprocess) i used a queue filled it with file names, opened a thread separately, read from it and then waited till the entire reading was over, after which i used to rename them. In this way i used to read files in multithreaded(in a batch).

Now, i want to read the files in a directory using both multiprocess and multithreads. I tried merging my two approaches but that didn't fare well. log showed a lot of files were showing FileNotFound exception(because their names were changed), some were never read (because thread died), sometimes locks were not released.

 ///////////////////////////////////////////////////////////////////////
     //file filter inner class
     class myfilter implements FileFilter{

            @Override
            public boolean accept(File pathname) {
                // TODO Auto-generated method stub
                Pattern pat = Pattern.compile("email[0-9]+$");
                Matcher mat = pat.matcher(pathname.toString());
                if(mat.find()) {
                    return true;
                }
                return false;
            }

        }
     /////////////////////////////////////////////////////////////////////////


    myfilter filter = new myfilter();
    File alreadyread[] = new File[5];
    Thread t[] = new Thread[5];
    fileread filer[] = new fileread[5];
    File file[] = directory.listFiles(filter);
    FileChannel filechannel[] = new FileChannel[5];
    FileLock lock[] = new FileLock[5];
    tuple_json = new ArrayList();
    //System.out.println("ayush");
    while(true) {
        //declare a queue
        ConcurrentLinkedQueue filequeue = new ConcurrentLinkedQueue();

        //addfilenames to queue and their renamed file names
        try{
        if(file.length!=0) {
            //System.out.println(file.length);
            for(int i=0;i<5 && i<file.length;i++) {

                System.out.println("acquiring lock on file " + file[i].toString());
                try{
                filechannel[i] = new RandomAccessFile(file[i], "rw").getChannel();
                lock[i] = filechannel[i].tryLock();
                }
                catch(Exception e) {
                    file[i] = null;
                    lock[i] = null;
                    System.out.println("cannot acquire lock");
                }
                if(lock[i]!=null){
                    System.out.println("lock acquired on file " + file[i].toString());
                   filequeue.add(file[i]);
                   alreadyread[i] = new File(file[i].toString() + "read");
                   System.out.println(file[i].toString() + "-----" + times);
                }
                else{

                   System.out.println("else condition of acquiring lock");
                    file[i] = null;
                }
                System.out.println("-----------------------------------");
            }

        //starting the thread to read the files
        for(int i=0;i<5 && i<file.length && lock[i]!=null && file[i]!=null;i++){
            filer[i] = new fileread(filequeue.toArray()[i].toString());
            t[i] = new Thread(filer[i]);
            System.out.println("starting a thread to read file" + file[i].toString());
            t[i].start();
        }

        //read the text
        for(int i=0;i<5 && i<file.length && lock[i]!=null && file[i]!=null;i++) {
            try {
                System.out.println("waiting to read " + file[i].toString() + " to be read completely");
                t[i].join();
                System.out.println(file[i] + " was read completetly");
                //System.out.println(filer[i].getText());

            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

        }
        //file has been read Now rename the file
        for(int i=0;i<5 && i<file.length && lock[i]!=null && file[i]!=null;i++){
            if(lock[i]!=null){
                System.out.println("renaming file " + file[i].toString());
                file[i].renameTo(alreadyread[i]);
                System.out.println("releasing lock on file  " + file[i].toString());
                lock[i].release();
        }
        }

//rest of the processing
/////////////////////////////////////////////////////////////////////////////////////////////////////

Fileread class

class fileread implements Runnable{
//String loc = "/home/ayusun/workspace/Eclipse/fileread/bin";
String fileloc;
BufferedReader br;
String text = "";
public fileread(String filename) {
    this.fileloc = filename;
}
@Override
public void run() {
    try {
        br = new BufferedReader(new FileReader(fileloc));
        System.out.println("started reading file" + fileloc);
        String currline;
        while((( currline = br.readLine())!=null)){
            if(text == "")
                text += currline;
            else
                text += "\n" + currline;
        }

        System.out.println("Read"  + fileloc  + " completely");
        br.close();

    } catch ( IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

}

public String getText() {
    return text;
}
}

I would like to know, if there is nay other approach that i can adopt.

4

2 回答 2

4

如果要创建对文件的独占访问，则不能使用文件锁定，因为在大多数操作系统上，文件锁定是建议性的，而不是强制性的。

我建议为所有进程创建一个通用锁定目录；在此锁定目录中，您将在打开文件之前为每个要锁定的文件创建一个目录。

最大的优点是目录创建与文件创建不同，是原子的；因此，您可以使用Files.createDirectory()（或 File .mkdir()，如果您仍然使用 Java6 但不要忘记检查返回码）来锁定您阅读的文件。如果失败，您知道其他人正在使用该文件。

当然，当你完成一个文件时，不要忘记删除与这个文件匹配的锁定目录...（在一个finally块中）

（注意：您可以使用 Java 7 Files.newBufferedReader()；甚至还有Files.readAllLines()）

于 2013-06-17T09:10:21.357 回答
0

如果您需要使用多个线程处理大量文件，您可能应该先将特定文件分配给每个线程，然后再开始。

例如，如果您只想处理名称以数字开头email并后跟一些数字的文件，则可以创建 10 个线程。第一个线程将查找名称以 开头的文件email0，第二个线程可以处理email1等等。

这当然只有在数字均匀分布的情况下才会有效。

另一种方法是让主线程运行并收集所有文件名来处理。然后，它可以将文件划分为可用线程的数量，并向每个线程传递一个包含这些文件名的数组。

可能有其他分配与您的情况相关的系统负载的方法。

于 2013-06-17T09:17:40.100 回答