Skip to content

Sort operation used more than the maximum 33554432 bytes of RAM #8

@bodqhrohro

Description

@bodqhrohro

I encountered this when exporting a large chat (137988 messages):

Traceback (most recent call last):
  File "/media/d/temp/git/matrix-archive/export_messages.py", line 80, in <module>
    export_archive()
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/media/d/temp/git/matrix-archive/export_messages.py", line 66, in export_archive
    print(f"Writing {len(messages)} messages to {filename!r}")
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 62, in __len__
    list(self._iter_results())
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 110, in _iter_results
    self._populate_cache()
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/queryset.py", line 129, in _populate_cache
    self._result_cache.append(next(self))
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/mongoengine/queryset/base.py", line 1619, in __next__
    raw_doc = next(self._cursor)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/cursor.py", line 1207, in next
    if len(self.__data) or self._refresh():
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/cursor.py", line 1124, in _refresh
    self.__send_message(q)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/cursor.py", line 999, in __send_message
    response = client._run_operation_with_response(
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1368, in _run_operation_with_response
    return self._retryable_read(
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
    return func(session, server, sock_info, slave_ok)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1360, in _cmd
    return server.run_operation_with_response(
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/server.py", line 135, in run_operation_with_response
    _check_command_response(first, sock_info.max_wire_version)
  File "/home/bodqhrohro/.local/lib/python3.8/site-packages/pymongo/helpers.py", line 160, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit., full error: {'ok': 0.0, 'errmsg': 'Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.', 'code': 96, 'codeName': 'OperationFailed'}

This can be worked around, though adding an actual index should be better.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions