As said before, the library will not be able to know how the application (being client to the library) is compiled, but the other way round has to work. Besides, I think you are talking about dynamic linking, since static linking certainly would not have different switches at same build time.
Similar to the already given answer by "Andrew Johnson", the library could provide a method for finding out whether it was compiled with large file support or not. Knowing that such build-time switches are mostly done with defines in C, this could be looking like this:
//in library:
BOOL isLargeFileSupport (void)
{
#ifdef LARGE_FILE_SUPPORT
return TRUE;
#else
return FALSE;
#endif
}
The application then knows how to handle file sizes reported by that lib, or can refuse to work when incompatible:
//in application
BOOL bLibLFS = lib_isLargeFileSupport();
BOOL bAppLFS = FALSE;
#ifdef LARGE_FILE_SUPPORT
bAppLFS = TRUE;
#endif
if (bLibLFS != bAppLFS)
//incompatible versions, bail out
exit(0);