Separate indexed datafiles into subdirectories of dataset and calendar months

[moeyensj]

We are working to support multiple different datasets, however, as it stands indexing multiple datasets places them all in the same set of binary files. Ideally, we'd want to be able to subsample the data so we can combine different dataset indexes or split them apart trivially.

The solution would be to separate each dataset's observations into their own subdirectory and then separate each binary file by calendar month. This way downsampling the full dataset becomes a simple table query.

Proposed schema:

$INDEX_DIR
- index.db
- data
  - nsc_dr2
    - 2012-10-01T00:00:00.000_2012-10-31T23:59:59.999.data
    - 2012-11-01T00:00:00.000_2012-11-30T23:59:59.999.data
    - ...
  - sdss_dr9
    - 2012-10-01T00:00:00.000_2012-10-31T23:59:59.999.data
    - 2012-11-01T00:00:00.000_2012-11-30T23:59:59.999.data
    - ...

I've modified a code snippet from Stack Overflow that can be used to divide into calendar months (this could be a good starting place for any implementation).

# Modified from: https://stackoverflow.com/questions/51293632/how-do-i-divide-a-date-range-into-months-in-python
import datetime
import numpy as np
from astropy.time import Time

begin = '2012-09-01'
end = '2020-01-01'

dt_start = datetime.datetime.strptime(begin, '%Y-%m-%d')
dt_end = datetime.datetime.strptime(end, '%Y-%m-%d')
one_day = datetime.timedelta(1)
start_dates = [dt_start]
end_dates = []
today = dt_start
while today <= dt_end:
    tomorrow = today + one_day
    if tomorrow.month != today.month:
        start_dates.append(tomorrow)
        # End Date is 1 millisecond before midnight of the first day of the next month
        end_dates.append(today + datetime.timedelta(1 - 1/86400/1000))
    today = tomorrow

end_dates.append(dt_end)

start_times = Time(start_dates)
end_times = Time(end_dates)
start_times = start_times[:-2]
end_times = end_times[:-2]

for i, j in zip(start_times.isot, end_times.isot):
    print(i, j)

start_times and end_times are both astropy Time objects that can then be used to filter and appropriate split the input observations.
The loop prints the following:

2012-09-01T00:00:00.000 2012-09-30T23:59:59.999
2012-10-01T00:00:00.000 2012-10-31T23:59:59.999
2012-11-01T00:00:00.000 2012-11-30T23:59:59.999
2012-12-01T00:00:00.000 2012-12-31T23:59:59.999
2013-01-01T00:00:00.000 2013-01-31T23:59:59.999
2013-02-01T00:00:00.000 2013-02-28T23:59:59.999
2013-03-01T00:00:00.000 2013-03-31T23:59:59.999
2013-04-01T00:00:00.000 2013-04-30T23:59:59.999
2013-05-01T00:00:00.000 2013-05-31T23:59:59.999
2013-06-01T00:00:00.000 2013-06-30T23:59:59.999
2013-07-01T00:00:00.000 2013-07-31T23:59:59.999
2013-08-01T00:00:00.000 2013-08-31T23:59:59.999
2013-09-01T00:00:00.000 2013-09-30T23:59:59.999
2013-10-01T00:00:00.000 2013-10-31T23:59:59.999
2013-11-01T00:00:00.000 2013-11-30T23:59:59.999
2013-12-01T00:00:00.000 2013-12-31T23:59:59.999
2014-01-01T00:00:00.000 2014-01-31T23:59:59.999
2014-02-01T00:00:00.000 2014-02-28T23:59:59.999
2014-03-01T00:00:00.000 2014-03-31T23:59:59.999
2014-04-01T00:00:00.000 2014-04-30T23:59:59.999
2014-05-01T00:00:00.000 2014-05-31T23:59:59.999
2014-06-01T00:00:00.000 2014-06-30T23:59:59.999
2014-07-01T00:00:00.000 2014-07-31T23:59:59.999
2014-08-01T00:00:00.000 2014-08-31T23:59:59.999
...