multilingual vs monolingual corpora?

[chirila]

I'm thinking of using polyglotdb with a multilingual corpus (that is, doing comparative work across multiple languages, though most files have a single language each). I'm trying to work out if it would be better to have each language as its own database, or a single large database. I'm leaving towards multiple databases with combining data after export, since the enrichment rules might be different for different languages. However, if others have thought through this question and come to a difference answer that would be helpful!

[mmcauliffe]

I would generally recommend separate databases, since there wouldn't be any real benefit internal to polyglot algorithms leveraging multiple languages. Basically each corpus is going to represent its own contained unit of work that wouldn't benefit from including all data, so generating a CSV for one language at a time can help with some debugging if one of the corpora throws an error along the way.

There might be some benefit for doing some sanity checks in a more interactive way (i.e., through a jupyter notebook), but once there's just a script you have defined in the general way, you can split it up into separate databases. Having multiple languages might slow down some queries as well (there are indexes to speed things up, but less data in an index still does have benefits).

[msonderegger]

@massimolipari is using polyglotdb and should have thoughts.

[massimolipari]

Sorry, I missed this issue earlier! I would definitely recommend having separate databases for each language, precisely for the reason you mentioned: enrichment will look quite different from one language to another. One case that comes to mind is syllabification--you probably want the set of possible onsets to differ from one language to another. Also, some languages will require extra enrichment steps which won't make sense for other languages (like tone enrichment).

If you need to batch process languages, a good way to do that is to write a script that pulls the necessary information for each corpus (the path to the corpus, the vowel set, the location of enrichment files, etc.) from a corpus-specific configuration file. Somebody originally came up with this idea for working with SPADE data (various disparate corpora of English dialects), and an example of this can be found here (and elsewhere on the same repo).

I also have some examples of YAML files and scripts for running analyses on corpora of different languages for a paper I'm currently working on. These will one day (hopefully soon!) live on an OSF repo with some documentation, but for now I'd be happy to share these privately--just shoot me an email!