Normalising categories #8
Description
Looking at the list of categories, it would be useful to clean these facets. For example, entries include:
Remunerated employment, office, profession etc.
Remunerated employment, office, profession etc..
Remunerated employment, office, profession et
Remunerated employment, office, profession, etc...
Remunerated employment, office, profession etc
Remunerated employment, office, profession, etc
Remunerated employment, office, profession etc...
Remunerated employment, office, profession etc
Remunerated employment, office, profession, etc..
Remunerated employment, office, profession etc
Remunerated employment, office, profession, etc.
De-punc and stripping the etc? would normalise all a lot of cases.
The Overseas visit/Overseas visits facet also appears in multiple variants with differing amounts of leading and trailing whitespace, so leading/trailing space stripping would be another really effective and simple rule.
I thought the type column might be a normalised column, but it doesn't seem to be? Eg type 9 is predominantly related to shareholdings, and then there is Family members employed and paid from parliamentary expenses as part of the group?
I wonder: is there a way to easily define cleaning rules using sqlite_utils that could be applied to a column to create a _cleaned version of it?