While developing a corpus, data from different domains and sources is used and it is very rare that you find exact propostion of data from these different domains/sources. So instead of just collecting and dumping data, it is better to use sampling technique to adequately split the data into train, test and dev set. This python script can be used to split your data (collected from different sources) into train, test and dev/validation set using stratified random sampling.
- Moodser Hussain
- COMSATS University Islamabad, Lahore Campus
- Email: [email protected]
- This script support both type of splits (1) Percentage and (2) Number
- Place all parallel files of English/Lang1 in 'en' directory and Urdu/Lang2 in 'ur' directory.
- Place this script in the root directory of 'en' and 'ur' folder.
- Name of files should end with language extention (e.g. all files should have extenstion .en that are placed in 'en' directory).
- During execution program will ask for split type (percentage/number) and values so provide these values according according to your requirements.
- It will automatically fetch files from both directories to generate parallel files for train, test and validation set.
- Dr. Rao Muhammad Adeel Nawab
- Dr. Muhammad Sharjeel
This script was used for English-Urdu langauge pair. If you are intending to use it any other language pair, you can change the extensions accordingly.