0

I need to index content of doc/docx/pdf files uploaded by users and use Solr (1.4.1) ExtractingRequestHandler component (817165) for that. If that matters, I don't request indexing from it - the component is always called with extractOnly parameter returning text content of the document only and not adding it to the index on its own straight away (the content is then added to the index "outside" as a text field of the document following the standard procedure).

However, some files are not parsed and the component returns 500 Internal Server Error with no other details provided. Of all *.doc files submitted by our users about 30% of them fail to parse.

It is not the problem with Solr load - the files that cannot be parsed are always the same if you parse the same list of them again and again. It is also not about their size - many of them are smaller than other ones parsed successfully. Apparently, it is not about peculiar formatting (or at least that is not obvious) - almost all documents that fail to parse have coloured fonts, tables and images but many of the ones parsed successfully also have the same.

All these files open in Word without any warnings or errors. If you save them as docx Solr starts parsing them correctly but re-saving them in the same doc format with the same content doesn't help. Still, if all the content is removed and replaced by some lorem ipsum text, then saved as doc, they become correct.

As the content replacing helps, it should be something with some elements used in the documents but there is no description on Tika Formats page telling in which cases parsing of the document fails.

I've uploaded a sample file which fails to be parsed in case if anyone is curious enough to try it (it is archived to prevent Windows Live from converting it into "online document").

Currently as a way around I use an ancient antiword utility to parse those *.doc on which Solr fails (and antiword parses them perfectly). Still, it is obviously a crutch and I wonder if anybody else is facing the same issue - I failed to google it so probably that's me doing something wrong.

Or, if that's a known problem, what could be more elegant ways to solve it (I don't like relying on antiword)?

4

1 回答 1

0

I'd try upgrading Tika if I were you

I've taken your sample file, and tried it with the latest version of Tika. Extracting to text works just fine, I see

LOREM IPSUM
Lorem ipsum dolor sit amet
------

Home Phone:           000000000

Work   :   00000000

(etc)

So I suspect it's an issue with older versions of POI+Tika which has now been fixed.

(If you're using a custom built copy of SOLR, then you may just need to bump up the Tika dependency in the pom and re-build, and maven will take care of it for you. Otherwise a newer SOLR should have a newer Tika in as standard)

于 2011-06-16T15:10:13.987 回答