Fix Item Duplicate Detection Across Datetime Indexes #575

bountx · 2026-01-08T15:06:48Z

Description:
When ENABLE_DATETIME_INDEX_FILTERING=true, items with different
datetime values are stored in different datetime-partitioned indexes. The bulk insert methods (bulk_async_prep_create_item, bulk_sync_prep_create_item) were using client.exists(index=alias) directly, which could fail to detect existing items stored in other datetime indexes.

Scenario that caused duplicates:

Item product-123 with datetime: 2024-01-15 is inserted into stac_items_collection_2024-01
POST same product-123 with datetime: 2024-06-20 - alias check misses it
Item is inserted into stac_items_collection_2024-06 - duplicate created

Solution
Extracted the duplicate check logic from async_prep_create_item into reusable helper methods that properly iterate through all indexes in the collection alias:

_check_item_exists_in_collection (async)
_check_item_exists_in_collection_sync (sync)

Updated all three prep methods to use these helpers:

async_prep_create_item
bulk_async_prep_create_item
bulk_sync_prep_create_item

PR Checklist:

Code is formatted and linted (run pre-commit run --all-files)
Tests pass (run make test)
Documentation has been updated to reflect changes, if applicable
Changes are added to the changelog

jonhealy1

Looks good, just a couple of things

stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py

…calls

jonhealy1

@bountx Thanks looks great

CHANGELOG.md

stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py

stac_fastapi/tests/database/test_utils.py

YuriZmytrakov · 2026-01-14T12:48:48Z

stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py

+        alias_exists = await self.client.indices.exists_alias(name=alias)
+
+        if alias_exists:
+            return await check_item_exists_in_alias(self.client, alias, doc_id)


Optional, but would be great. I can see that its raising ConflictError, please consider adding a custom error that would have a relevant message error that this item already exists to improve the error quality.

stac_fastapi/tests/extensions/test_bulk_transactions.py

YuriZmytrakov · 2026-01-14T13:03:48Z

stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py

+            raise ConflictError(
+                f"Item {item['id']} in collection {item['collection']} already exists"
+            )


Please consider improving the error to return not only message, but more information such as which item ids are duplicated and in which conllection. Then you can assert easier the error in tests to check if the conflicted item id == the expected item id that is a duplicate:

raise ConflictError(
f"Item {item['id']} in collection {item['collection']} already exists",
item_id=item["id"],
collection=item["collection"],
)

…tence and duplicates

stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py

YuriZmytrakov · 2026-01-15T10:17:27Z

stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py

+            raise ConflictError(
+                f"Item {item['id']} in collection {item['collection']} already exists"
+            )


Looks like the custom error could be applied here as well as in other ConflictError cases.

Created extension of ConflicError class - ItemAlreadyExistsError

stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database/utils.py

stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py

…n item creation

YuriZmytrakov · 2026-01-15T15:54:21Z

stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database/utils.py

+        super().__init__(message)
+
+
+async def check_item_exists_in_alias(client: Any, alias: str, doc_id: str) -> bool:


Optional, please, consider using stricter type hints:
client: Union[AsyncElasticsearch, AsyncOpenSearch]

YuriZmytrakov · 2026-01-15T15:54:40Z

stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/database/utils.py

+    return bool(resp["hits"]["total"]["value"])
+
+
+def check_item_exists_in_alias_sync(client: Any, alias: str, doc_id: str) -> bool:


The same here.

…ages

…response messages

bountx · 2026-01-27T12:04:04Z

@jonhealy1 I added small fixes that popped up during dev tests and they pass now

jonhealy1

Changes look good nice work

bountx · 2026-01-29T09:39:17Z

@jonhealy1 ready to be merged!

Andrzej Pijanowski added 3 commits January 8, 2026 16:01

feat: duplicate validation across collection for POST requests

a622551

chore: formatting

8376ec7

chore: update CHANGELOG

3e3b7a2

bountx marked this pull request as ready for review January 8, 2026 15:12

bountx requested review from Gomez324, YuriZmytrakov and jonhealy1 and removed request for Gomez324 January 8, 2026 15:12

bountx self-assigned this Jan 8, 2026

jonhealy1 requested changes Jan 9, 2026

View reviewed changes

stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py Outdated Show resolved Hide resolved

stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py Show resolved Hide resolved

Andrzej Pijanowski added 3 commits January 12, 2026 11:17

fix: optimized existence check across collection to do it in O(1) DB …

4eef8ba

…calls

Merge branch 'main' into CAT-1643

74c1869

chore: pre-commit fixes

0a8ee1b

bountx requested a review from jonhealy1 January 12, 2026 11:17

jonhealy1 previously approved these changes Jan 14, 2026

View reviewed changes

YuriZmytrakov reviewed Jan 15, 2026

View reviewed changes

fix: improve error handling and add assertions in tests for item exis…

4b09e40

…tence and duplicates

bountx dismissed jonhealy1’s stale review via 4b09e40 January 15, 2026 10:03

fix: correct item ID formatting in existence checks and assertions

f644c4d

YuriZmytrakov reviewed Jan 15, 2026

View reviewed changes

stac_fastapi/opensearch/stac_fastapi/opensearch/database_logic.py Outdated Show resolved Hide resolved

Andrzej Pijanowski added 2 commits January 15, 2026 12:14

feat: introduce ItemAlreadyExistsError for better conflict handling i…

5ebdc2d

…n item creation

chore: change docstrings

83e52a8

bountx requested review from YuriZmytrakov and jonhealy1 January 15, 2026 14:46

YuriZmytrakov reviewed Jan 15, 2026

View reviewed changes

YuriZmytrakov previously approved these changes Jan 15, 2026

View reviewed changes

jonhealy1 previously approved these changes Jan 15, 2026

View reviewed changes

bountx and others added 2 commits January 15, 2026 18:11

Merge branch 'main' into CAT-1643

7cbe4cf

fix: handle duplicate items in bulk operations and update return mess…

360aa42

…ages

bountx dismissed stale reviews from jonhealy1 and YuriZmytrakov via 360aa42 January 27, 2026 08:58

Andrzej Pijanowski added 3 commits January 27, 2026 10:05

Merge branch 'main' into CAT-1643

05d5902

fix: update validation message for duplicate items in bulk insert test

89e8d87

fix: enhance duplicate handling in bulk insert operations and update …

49ea220

…response messages

bountx requested a review from jonhealy1 January 27, 2026 12:12

Merge branch 'main' into CAT-1643

dc99f8a

jonhealy1 approved these changes Jan 28, 2026

View reviewed changes

bountx requested review from Gomez324 and YuriZmytrakov January 28, 2026 12:34

Gomez324 approved these changes Jan 29, 2026

View reviewed changes

jonhealy1 merged commit c81fa6b into main Jan 29, 2026
8 checks passed

jonhealy1 deleted the CAT-1643 branch January 29, 2026 10:59

bountx restored the CAT-1643 branch January 29, 2026 12:07

bountx deleted the CAT-1643 branch January 29, 2026 12:10

bountx restored the CAT-1643 branch January 29, 2026 13:28

bountx deleted the CAT-1643 branch January 29, 2026 13:31

		super().__init__(message)


		async def check_item_exists_in_alias(client: Any, alias: str, doc_id: str) -> bool:

		return bool(resp["hits"]["total"]["value"])


		def check_item_exists_in_alias_sync(client: Any, alias: str, doc_id: str) -> bool:

Fix Item Duplicate Detection Across Datetime Indexes #575

Fix Item Duplicate Detection Across Datetime Indexes #575

Uh oh!

Conversation

bountx commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YuriZmytrakov Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bountx Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

YuriZmytrakov Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

bountx Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

YuriZmytrakov Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

bountx Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YuriZmytrakov Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YuriZmytrakov Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

bountx commented Jan 27, 2026

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

bountx commented Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bountx commented Jan 8, 2026 •

edited

Loading

YuriZmytrakov Jan 14, 2026 •

edited

Loading

YuriZmytrakov Jan 15, 2026 •

edited

Loading