c# - Will a transaction always commit (making sure file is imported only once)?

Question

The question I ask needs some introduction. I have an application that imports small files and e-mails into its own SQL Server database from a folder every 5 minutes. It is of great importance that

1. each file must be imported
2. each file must only be imported once

Now part 1 is not a big deal. Part 2 however raises some concerns. If the file cannot be deleted after being successfully imported (for example due to lack of access rights), and the next time my application looks at the folder the file will be imported again and again, until someone fixes the access rights problem.

Thus I was wondering if the following approach would work safely, without data loss:

1. start a transaction
2. import the file into the DB
3. delete the file from the folder
4. commit the transaction (or rollback if there was an error)

This is the point where I arrived to the title of the question: will the last commit step always succeed if prior steps did not throw an exception? Am I safe to delete the file before the commit, without risking losing the file? What if SQL Server is shut down during transaction? Or shall I delete the record from the database if file deletion failed?

Answer 1

The "transaction" you speak of would only be for ensuring the data was fully committed to the database. To add durability to ensure that the file can't be imported more than once, you have an enumerable number of options, but let's consider the following:

1. When importing the file, add the file name to a table in which that is a unique constraint so that the SQL INSERT would fail and thus the entire "transaction" rolled back. This of course will only work if the file names are in fact unique. If the file name isn't unique, then see if you can find enough information that is. Often times there is additional domain information you can put with it (e.g. invoice #, customer #, etc.) to make it unique and thus the INSERT still fail.
2. Keep a listing of all the files that have been processed and don't process them again. You could do this with an XML file, or even in memory, but it's still risky because if the user's drop a new file in there with the same name - well you actually want to import that one. I think Option 1 is about the best bet here.

One thing you need to realize though is that you have two separate technologies you're managing. The deletion of the file, or should I say your attempt, doesn't have anything to do with the database transaction unless you make it. So, let's say you had a flow like this:

using (SqlConnection c ...)
using (SqlCommand cmd ...)
{
    SqlTransaction t = c.BeginTransaction();
    try
    {
        // update database

        // try to delete file

        t.Commit();
    }
    catch (Exception ex)
    {
        // you know something failed
        // you can catch more specific exceptions and respond here
    }
}

If the t.Commit() isn't reached, the database will not be updated.

Answer 2

Well generally I would first archive the file to another location. Then I would set up a place to store meta data abaout the imports, that would store the filename and the date of the file and the success or failure of the import. Then you could in fact prove that each file had been imported and would still have a copy of any that failed.

Then I would run the process in a transaction to import the file. I would put the meta data about the file into the database before starting the import transaction (so it won;t rollback on failure) and then after the commit, I would add information to the meta data to let me mknow the file had been successfully imported. Of course being a data person, I would do all this in SSIS where I could use the native logging to help me document each step (helps to see where a failure occurred). After teh import completed I would delete the file from teh processing location, but I would keep it for a set period of time in the archive location. (There are alawys later questions about why the data looks teh way it does that it helps to have the actual file on hand to show them they sent it to you that way!)

Answer 3

Assuming you are using a relatively recent version of windows (Vista+) and are using NTFS, you can enlist file operations in a transaction.

I don't believe there is support for this built in to the .NET libraries yet, but there is a third party library here that would let you do something like this:

// Wrap a file copy and a database insert in the same transaction
TxFileManager fileMgr = new TxFileManager();
using (TransactionScope scope1 = new TransactionScope())
{
    // Delete the file
    fileMgr.DeleteFile(fileName);

    // Do some database stuff
    dbMgr.ExecuteNonQuery(insertSql);

    scope1.Complete();
}