Enable custom Tika Parser #498

jevertz · 2018-02-02T14:53:58Z

This makes it possible to use a custom Tika Parser.
To use the parser, just an addition to _settings.json is necessary.
Here's an example

    "custom_tika_parsers": [
      {
        "class_name": "com.monotype.OTParser",
        "path_to_jar": "/Users/jevertz/workspace/testfscrawler/svn-search/customTikaParsers/target/otparser-0.0.1-SNAPSHOT.jar",
        "mime_types": ["application/x-font-ttf", "application/x-font-otf"]
      }
    ]

The jar file does not need to be on the class path.
The parser needs to be a Tika parser, like described here

dadoonet · 2018-02-02T17:07:03Z

Very interesting. I like the idea.

I wonder if it's absolutely needed to provide the tika-config.xml file BTW.

I'll give a proper review later but thanks for sharing your code with this PR!

jevertz · 2018-02-03T10:26:36Z

You’re. welcome. Actually I implemented first passing a Tika Config which can be useful, but not for my use-case. So I took a different approach and removed the need for a Tika Config file later. Best Jochen Evertz

dadoonet · 2018-02-15T22:47:59Z

Hi @jevertz

Would you mind rebasing your code on latest master branch?

dadoonet · 2018-02-15T22:48:44Z

Also I think it would be good to document that change in the README.

jevertz · 2018-02-19T16:32:56Z

Hi David, might take a little since the ES version changed. Don’t want to break my setup. But not weeks. Best Jochen

dadoonet

That looks very promising. Thanks!

I added a lot of "format" comments.
In the README, I'm always trying to print a full setting configuration in https://github.com/dadoonet/fscrawler#job-file-specification
May be you could add your changes as well here. IIRC I'm using part of the output of the FsSettingsParserTest execution.

dadoonet · 2018-02-22T16:23:52Z

README.md

+It might occur that one or more existing Tika parsers do not provide the intended information, or just do not exist.
+This setting allows to use a custom parser instead.
+The parsers must be provided as a .jar, but does not need to be on any classpath.
+Note that this is an array. Here an example for just one


NIT:

Here an example for just one

to

Here an example for just one:

dadoonet · 2018-02-22T16:24:55Z

README.md

+
+Some info about creating a custom parser is available [here](https://tika.apache.org/1.17/parser_guide.html)
+Or use a existing parser as a blueprint. Make sure to choose the correct branch.
+At the time of this writing fscrawler uses Tika 1.17, while on github the main branch is 2.x.


the main branch is 2.x

to

the main Tika branch is 2.x

dadoonet · 2018-02-22T16:26:24Z

README.md

+The parsers from ["branch_1x"](https://github.com/apache/tika/tree/branch_1x/tika-parsers/src/main/java/org/apache/tika/parser) should work fine.
+
+To build the custom parser separately, a pom file can be derived from the tika-parsers [pom.xml](https://github.com/apache/tika/blob/branch_1x/tika-parsers/pom.xml).
+Probably a lot can be left out. Here is an example which required fontbox (guess still to long, but worked).


WDYM with "guess still to long, but worked"

too long? Is it a comment you are making here? If so I believe this is not needed.

Just to indicate that I didn’t really evaluated the Pom.xml, but stopped when it build without errors.
Should get marked someway as to-do

Changed it to
(The exclusions are copied 1:1 from fscrawler's pom.xml, to be on the safe side)

dadoonet · 2018-02-22T16:27:25Z

README.md

+                    <groupId>com.uwyn</groupId>
+                    <artifactId>jhighlight</artifactId>
+                </exclusion>
+                <!-- ES core already has these -->


ES Core -> FSCrawler Core ?

I copied the excludes from fscrawler's root pom.xml. The line is still there at the time of this writing. Can it be removed, perhaps?

That's true. Just remove it here but I will also remove this in the future from the main pom.xml indeed.

dadoonet · 2018-02-22T16:28:32Z

README.md

+    <version>0.0.1-SNAPSHOT</version>
+
+    <properties>
+        <maven.compiler.source>1.7</maven.compiler.source>


Not an issue but I'd recommend using 1.8 as the version. I know that 1.7 will work but this JVM is no longer supported so no need to advertise it :)

You’re completely right. I just didn’t notice the setting.
I’ll change it to 1.8

dadoonet · 2018-02-22T16:34:22Z

settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/Fs.java

+        }
+
+        public Builder addTikaCustomParsers(CustomTikaParser customTikaParser) {
+            if (this.customTikaParsers == null) {


I believe this can not normally happen?

dadoonet · 2018-02-22T16:34:52Z

settings/src/test/java/fr/pilato/elasticsearch/crawler/fs/settings/FsSettingsParserTest.java

@@ -53,6 +54,7 @@
            .setUpdateRate(TimeValue.timeValueMinutes(5))
            .setIndexContent(true)
            .setOcr(OCR_FULL)
+            .setTikaCustomParsers(new ArrayList<>())


I think you should add at least a singleton List of one CustomTikaParser element here to make sure serialisation and deserialisation is working well.

dadoonet · 2018-02-22T16:35:09Z

tika/src/main/java/fr/pilato/elasticsearch/crawler/fs/tika/TikaInstance.java

+                        Parser customParserDecorated = ParserDecorator.withTypes(customParser, customMediaTypes);
+                        PARSERS[counter] = customParserDecorated;
+
+


dadoonet · 2018-02-22T16:37:27Z

tika/src/main/java/fr/pilato/elasticsearch/crawler/fs/tika/TikaInstance.java

+
+
+
+                    } catch (IOException e) {


I think you can just do here something like:

} catch (IOException|ClassNotFoundException|InstantiationException|IllegalAccessException e) { logger.error("Caught {}: {}", e.getClass().getSimpleName(), e.getMessage()); }

Or just catch any Exception?

dadoonet · 2018-02-22T16:38:21Z

tika/src/main/java/fr/pilato/elasticsearch/crawler/fs/tika/TikaInstance.java

+                        logger.error("Caught InstantiationException:" + e.getMessage());
+                    } catch (IllegalAccessException e) {
+                        logger.error("Caught IllegalAccessException:" + e.getMessage());
+                    }/*catch (TikaException te) {


Probably we don't need that?

jevertz · 2018-02-23T20:40:18Z

You’re welcome! Sorry for the delay. I‘ll merge your corrections ASAP - what probably means next Monday. I wasn‘t sure whether it’s a good idea to include it in a full example since an error will be thrown if the custom jar doesn’t exist. Or do you think it is useful to add a complete custom parser? I didn’t do it since I thought it might confuse most users.

dadoonet · 2018-02-23T21:12:27Z

Ideally you would create a submodule which is added as a test dependency in the integration tests module.
So we have something that compiles and that is tested.

Then link in the README to the module and the test class.

WDYT?

jevertz · 2018-02-23T22:04:57Z

OK, I‘ll see to it. Might be clearer to use an example with a custom extension instead of mime-type. Have to look into that. Do you perhaps know a starting point to do that when setting up a Tika parser from FSCrawler?

dadoonet · 2018-02-25T18:10:07Z

Do you perhaps know a starting point to do that when setting up a Tika parser from FSCrawler?

I'm confused. Isn't what you proposed with that:

{
  "name": "test",
  "fs": {
    "custom_tika_parsers": [
      {
        "class_name": "org.me.MyParser",
        "path_to_jar": "/some/full/path/to/myParser-0.0.1-SNAPSHOT.jar",
        "mime_types": ["application/dns", "or-another-mimetype-from-tika"]
      }
    ]
  }
}

I mean that you could create fscrawler-sample-parser.jar which you can add in a test?
If you don't know how to do it, no worries, I can do it in a follow up PR.

jevertz · 2018-02-26T11:13:29Z

Sorry, for causing this confusion. All my fault. That was expressed bad from me. At the moment I use the mime_types. However there are files which do not have a mime type, but a custom extension. That means I need to pass a custom mime type to Tika. Thought you might have something like that before, so I dared to asked. I’ll just see how to accomplish it. The documentation of Tika<https://tika.apache.org/1.17/configuring.html> is a little brief in this regard: Configuring Mime Types TODO Mention non-standard paths, and custom mime type files So I dared to ask whether you might encountered this and give me a hint were to look. Probably the decorators, but I’ll find out Best Jochen

https://github.com/dadoonet/fscrawler/pull/498/files/224f22e5611acb7308ac2cbe25d6a22602894773

# Conflicts: # README.md # settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/CustomTikaParser.java # settings/src/main/java/fr/pilato/elasticsearch/crawler/fs/settings/Fs.java # settings/src/test/java/fr/pilato/elasticsearch/crawler/fs/settings/FsSettingsParserTest.java # tika/src/main/java/fr/pilato/elasticsearch/crawler/fs/tika/TikaInstance.java

dadoonet · 2018-03-01T04:22:24Z

Not sure what is happening with your branch but I can see a lot of changes which are not related to your PR.
May be you could rebase on master and while we are at it squash the changes?

jevertz · 2018-03-01T21:12:05Z

Sorry,I messed up the rebase. I‘ll see to it to correct it. We used perforce before, so I can only apologize for making noob git mistakes.

dadoonet · 2018-07-07T16:04:56Z

@jevertz Do you think you can come back with an updated PR at some point?

jevertz · 2018-07-30T09:11:20Z

I'll see to do the rebase. Sorry for the delay, I was on holiday.
Probably It's best to include the changes manually again. To which version do you like me to base upon?

dadoonet · 2018-07-30T09:36:36Z

Sorry for the delay, I was on holiday.

Sure. No problem.

Probably It's best to include the changes manually again. To which version do you like me to base upon?

Why not. You can close this PR and open a new one.
It must be done on master branch.

Let me know if you need help.

notenbrood · 2019-04-09T13:42:23Z

@jevertz @dadoonet Sorry for the revive, but this feature would actually be pretty interesting to us. Are you still planning on finishing this PR?

janhoy · 2020-09-06T00:19:22Z

See #1004 for a related effort, which will let users take full control of what happens with a file before it gets indexed.
It could help solve this usecase, perhaps not as elegant by config but by giving you a hook to plug in your Java code.

dadoonet · 2022-05-10T13:56:41Z

This is now supported I think with #1367

dadoonet added this to the 2.5 milestone Feb 15, 2018

dadoonet added the new For new features or options label Feb 15, 2018

jevertz added 4 commits February 21, 2018 11:27

Try to resolve conflict before overwrite from origin

2ae845f

Example Tika configuration file.

28482c8

allow custom Tika parser

f7e2b1a

After rebase. Added support for a custom external Tika Parser.

1a0f5f6

jevertz force-pushed the master branch from 4289119 to 1a0f5f6 Compare February 21, 2018 14:23

Extended README to cover custom tika parsers setting

224f22e

dadoonet self-requested a review February 22, 2018 13:46

dadoonet self-assigned this Feb 22, 2018

dadoonet requested changes Feb 22, 2018

View reviewed changes

jevertz added 7 commits February 28, 2018 14:54

Try to resolve conflict before overwrite from origin

5c4b88d

Example Tika configuration file.

43e42aa

allow custom Tika parser

4e9cd5b

After rebase. Added support for a custom external Tika Parser.

456104c

Extended README to cover custom tika parsers setting

ed528b2

Implemented change requests

05a609a

https://github.com/dadoonet/fscrawler/pull/498/files/224f22e5611acb7308ac2cbe25d6a22602894773

dadoonet removed this from the 2.5 milestone Jul 7, 2018

dadoonet changed the title ~~Enable custom tiki Parser~~ Enable custom Tika Parser Jul 16, 2018

dadoonet mentioned this pull request Nov 25, 2019

Can we ignore parsing content at all by adding config key in settings? #846

Open

iadcode mentioned this pull request Jan 27, 2022

Add option to set path to custom tika config file #1367

Merged

dadoonet removed the new For new features or options label May 10, 2022

dadoonet closed this May 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable custom Tika Parser #498

Enable custom Tika Parser #498

jevertz commented Feb 2, 2018 •

edited by dadoonet

Loading

dadoonet commented Feb 2, 2018

jevertz commented Feb 3, 2018 via email •

edited by dadoonet

Loading

dadoonet commented Feb 15, 2018

dadoonet commented Feb 15, 2018

jevertz commented Feb 19, 2018 via email •

edited by dadoonet

Loading

dadoonet left a comment

dadoonet Feb 22, 2018

dadoonet Feb 22, 2018

dadoonet Feb 22, 2018

jevertz Feb 23, 2018

jevertz Feb 28, 2018

dadoonet Feb 22, 2018

jevertz Feb 28, 2018

dadoonet Mar 1, 2018

dadoonet Feb 22, 2018

jevertz Feb 23, 2018

dadoonet Feb 22, 2018

dadoonet Feb 22, 2018

dadoonet Feb 22, 2018

dadoonet Feb 22, 2018

dadoonet Feb 22, 2018

jevertz commented Feb 23, 2018 via email •

edited by dadoonet

Loading

dadoonet commented Feb 23, 2018

jevertz commented Feb 23, 2018 via email •

edited by dadoonet

Loading

dadoonet commented Feb 25, 2018

jevertz commented Feb 26, 2018 via email •

edited

Loading

dadoonet commented Mar 1, 2018

jevertz commented Mar 1, 2018 via email •

edited by dadoonet

Loading

dadoonet commented Jul 7, 2018

jevertz commented Jul 30, 2018

dadoonet commented Jul 30, 2018

notenbrood commented Apr 9, 2019

janhoy commented Sep 6, 2020

dadoonet commented May 10, 2022

		Parser customParserDecorated = ParserDecorator.withTypes(customParser, customMediaTypes);
		PARSERS[counter] = customParserDecorated;

Enable custom Tika Parser #498

Enable custom Tika Parser #498

Conversation

jevertz commented Feb 2, 2018 • edited by dadoonet Loading

dadoonet commented Feb 2, 2018

jevertz commented Feb 3, 2018 via email • edited by dadoonet Loading

dadoonet commented Feb 15, 2018

dadoonet commented Feb 15, 2018

jevertz commented Feb 19, 2018 via email • edited by dadoonet Loading

dadoonet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jevertz commented Feb 23, 2018 via email • edited by dadoonet Loading

dadoonet commented Feb 23, 2018

jevertz commented Feb 23, 2018 via email • edited by dadoonet Loading

dadoonet commented Feb 25, 2018

jevertz commented Feb 26, 2018 via email • edited Loading

dadoonet commented Mar 1, 2018

jevertz commented Mar 1, 2018 via email • edited by dadoonet Loading

dadoonet commented Jul 7, 2018

jevertz commented Jul 30, 2018

dadoonet commented Jul 30, 2018

notenbrood commented Apr 9, 2019

janhoy commented Sep 6, 2020

dadoonet commented May 10, 2022

jevertz commented Feb 2, 2018 •

edited by dadoonet

Loading

jevertz commented Feb 3, 2018 via email •

edited by dadoonet

Loading

jevertz commented Feb 19, 2018 via email •

edited by dadoonet

Loading

jevertz commented Feb 23, 2018 via email •

edited by dadoonet

Loading

jevertz commented Feb 23, 2018 via email •

edited by dadoonet

Loading

jevertz commented Feb 26, 2018 via email •

edited

Loading

jevertz commented Mar 1, 2018 via email •

edited by dadoonet

Loading