Skip to content

Conversation

@jeanconn
Copy link
Contributor

@jeanconn jeanconn commented Sep 10, 2025

Description

Update allowed size of acquisition database.

Because I used h5 badly when making the database, it grows I think via a leak mechanism. So I've got the job set to ask me to repack it when it gets too big. We could also auto-repack it or fix the underlying problem, but for now, I think it makes sense to increase the size where I get a warning, because the database when compressed is getting closer to the limit (it was 44Mb repacked today).

Interface impacts

Testing

Unit tests

  • Mac
(ska3-latest) flame:mica jean$ pytest
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir: /Users/jean/git
configfile: pytest.ini
plugins: socket-0.7.0, anyio-4.7.0, timeout-2.3.1
collected 113 items                                                                                                                                              

mica/archive/tests/test_aca_dark_cal.py ..................                                                                                                 [ 15%]
mica/archive/tests/test_aca_hdr3.py ..                                                                                                                     [ 17%]
mica/archive/tests/test_aca_l0.py ...ss                                                                                                                    [ 22%]
mica/archive/tests/test_asp_l1.py sssssss                                                                                                                  [ 28%]
mica/archive/tests/test_cda.py ..............................................                                                                              [ 69%]
mica/archive/tests/test_obspar.py .                                                                                                                        [ 69%]
mica/report/tests/test_report.py ss                                                                                                                        [ 71%]
mica/report/tests/test_write_report.py s                                                                                                                   [ 72%]
mica/starcheck/tests/test_catalog_fetches.py ...............                                                                                               [ 85%]
mica/stats/tests/test_acq_stats.py ...                                                                                                                     [ 88%]
mica/stats/tests/test_guide_stats.py ....                                                                                                                  [ 92%]
mica/vv/tests/test_vv.py ss.ssssss                                                                                                                         [100%]

======================================================================== warnings summary ========================================================================
mica/mica/archive/tests/test_cda.py::test_get_proposal_abstract
mica/mica/archive/tests/test_cda.py::test_get_proposal_abstract
mica/mica/starcheck/tests/test_catalog_fetches.py::test_validate_catalogs_over_range
mica/mica/starcheck/tests/test_catalog_fetches.py::test_validate_catalogs_over_range
  /Users/jean/miniforge3/envs/ska3-latest/lib/python3.12/site-packages/bs4/builder/_lxml.py:124: DeprecationWarning: The 'strip_cdata' option of HTMLParser() has never done anything and will eventually be removed.
    parser = parser(

mica/mica/starcheck/tests/test_catalog_fetches.py::test_validate_catalogs_over_range
  /Users/jean/miniforge3/envs/ska3-latest/lib/python3.12/site-packages/django/utils/encoding.py:266: DeprecationWarning: 'locale.getdefaultlocale' is deprecated and slated for removal in Python 3.15. Use setlocale(), getencoding() and getlocale() instead.
    encoding = locale.getdefaultlocale()[1] or 'ascii'

mica/mica/starcheck/tests/test_catalog_fetches.py::test_validate_catalogs_over_range
  /Users/jean/miniforge3/envs/ska3-latest/lib/python3.12/site-packages/django/http/request.py:1: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
    import cgi

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================== 93 passed, 20 skipped, 6 warnings in 61.71s (0:01:01) ======================================================
(ska3-latest) flame:mica jean$ git rev-parse HEAD
da047ca3dc618048ea8bdcf755543acebc428390

Independent check of unit tests by [REVIEWER NAME]

  • [PLATFORM]:

Functional tests

I just tested this with a local install and the current (just over 50MB database), showing that the flight code gives a warning and the new PR code does not.

(ska3-latest) flame:mica jean$ /Users/jean/miniforge3/envs/ska3-latest/share/mica/update_acq_stats.py 

Warning: /Users/jean/ska/data/acq_stats/acq_stats.h5 is larger than 50MB and may need
Warning: to be manually repacked (i.e.):
Warning:
Warning: ptrepack --chunkshape=auto --propindexes --keep-source-filters /Users/jean/ska/data/acq_stats/acq_stats.h5 compressed.h5
Warning: cp compressed.h5 /Users/jean/ska/data/acq_stats/acq_stats.h5


(ska3-latest) flame:mica jean$ git checkout acq-stats-size
Switched to branch 'acq-stats-size'
(ska3-latest) flame:mica jean$ pip install .
Processing /Users/jean/git/mica
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: mica
  Building wheel for mica (setup.py) ... done
  Created wheel for mica: filename=mica-4.39.1.dev3+gda047ca-py3-none-any.whl size=190723 sha256=f22df920f1d0d14fac8ca6404235cc81a451172997197d4f492b3032b576362e
  Stored in directory: /private/var/folders/f2/0_51vbvs2lg1cdt89x1lrdnr0000gp/T/pip-ephem-wheel-cache-ogpy1s6i/wheels/80/4f/05/60d1d1a86ebf793b23c1d072167f3c279dd4d69926d7c6cd76
Successfully built mica
Installing collected packages: mica
  Attempting uninstall: mica
    Found existing installation: mica 4.39.0
    Uninstalling mica-4.39.0:
      Successfully uninstalled mica-4.39.0
Successfully installed mica-4.39.1.dev3+gda047ca
(ska3-latest) flame:mica jean$ /Users/jean/miniforge3/envs/ska3-latest/share/mica/update_acq_stats.py 
(ska3-latest) flame:mica jean$

@jeanconn jeanconn merged commit 372558f into master Sep 18, 2025
2 checks passed
This was referenced Sep 18, 2025
@javierggt javierggt mentioned this pull request Oct 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants