BlockingPy is a Python package that implements efficient blocking methods for record linkage and data deduplication using Approximate Nearest Neighbor (ANN) algorithms. It is based on R blocking package.
Additionally, GPU acceleration is available via blockingpy-gpu (FAISS-GPU).
When performing record linkage or deduplication on large datasets, comparing all possible record pairs becomes computationally infeasible. Blocking helps reduce the comparison space by identifying candidate record pairs that are likely to match, using efficient approximate nearest neighbor search algorithms.
BlockingPy requires Python 3.10 or later. Installation is handled via PIP as follows:
pip install blockingpyYou may need to run the following beforehand:
sudo apt-get install -y libmlpack-dev # on Linux
brew install mlpack # on MacOSfor the GPU version: see here or docs
from blockingpy import Blocker
import pandas as pd
# Example data for record linkage
x = pd.DataFrame({
"txt": [
"johnsmith",
"smithjohn",
"smiithhjohn",
"smithjohnny",
"montypython",
"pythonmonty",
"errmontypython",
"monty",
]})
y = pd.DataFrame({
"txt": [
"montypython",
"smithjohn",
"other",
]})
# Initialize blocker instance
blocker = Blocker()
# Perform blocking with default ANN : FAISS
block_result = blocker.block(x = x['txt'], y = y['txt'])Printing block_result contains:
- The method used (
faiss- refers to Facebook AI Similarity Search) - Number of blocks created (
3in this case) - Number of columns (features) used for blocking (intersecting n-grams generated from both datasets,
17in this example) - Reduction ratio, i.e. how large is the reduction of comparison pairs (here
0.8750which means blocking reduces comparison by over 87.5%).
print(block_result)
# ========================================================
# Blocking based on the faiss method.
# Number of blocks: 3
# Number of columns created for blocking: 17
# Reduction ratio: 0.8750
# ========================================================
# Distribution of the size of the blocks:
# Block Size | Number of Blocks
# 2 | 3 By printing block_result.result we can take a look at the results table containing:
- row numbers from the original data,
- block number (integers),
- distance (from the ANN algorithm).
print(block_result.result)
# x y block dist
# 0 4 0 0 0.000000
# 1 1 1 1 0.000000
# 2 6 2 2 0.607768We can perform deduplication by putting previously created DataFrame in the block() method.
dedup_result = blocker.block(x = x['txt'])print(dedup_result)
# ========================================================
# Blocking based on the faiss method.
# Number of blocks: 2
# Number of columns created for blocking: 25
# Reduction ratio: 0.571429
# ========================================================
# Distribution of the size of the blocks:
# Block Size | Number of Blocks
# 4 | 2 print(dedup_result.result)
# x y block dist
# 0 1 0 0 0.125000
# 1 3 1 0 0.105573
# 2 1 2 0 0.105573
# 3 5 4 1 0.083333
# 4 4 6 1 0.105573
# 5 5 7 1 0.278312You can find more comprehensive examples in examples section.
-
Multiple ANN implementations available:
-
Multiple distance metrics such as:
- Euclidean
- Cosine
- Inner Product
and more...
-
Support for both shingle-based and embedding-based text representation
-
Comprehensive algorithm parameters customization with
control_annandcontrol_txt -
Support for already created Document-Term-Matrices (as
np.ndarrayorcsr_matrix) -
Support for both record linkage and deduplication
-
Evaluation metrics when true blocks are known
-
GPU support for fast blocking of large datasets usin GPU-accelerated indexes from FAISS
You can find detailed information about BlockingPy in documentation.
BlockingPy can process large datasets by utilizing the GPU with faiss_gpu algorithms. The available GPU indexes are (Flat/IVF/IVFPQ/CAGRA). blockingpy-gpu also includes all CPU indexes besides the mlpack backends.
- OS: Linux or Windows 11 with WSL2 (Ubuntu)
- Python: 3.10
- GPU: Nvidia with driver supporting CUDA ≥ 12.4
- Tools: conda/mamba + pip
PyPI wheels do not provide CUDA-enabled FAISS. You must install FAISS-GPU via conda/mamba, then install blockingpy-gpu with pip.
# 1) Env
mamba create -n blockingpy-gpu python=3.10 -y
conda activate blockingpy-gpu
conda config --env --set channel_priority flexible
# 2) Install FAISS GPU (nightly cuVS build) - this version was tested
mamba install -y \
-c pytorch/label/nightly -c rapidsai -c conda-forge \
"faiss-gpu-cuvs=1.11.0" "libcuvs=25.4.*"
# 3) Install BlockingPy and the rest of deps with pip (or poetry, uv etc.)
pip install blockingpy-gpuBlockingPy comes with example datasets fetched via Pooch library:
-
Census-Cis dataset created by Paula McLeod, Dick Heasman and Ian Forbes, ONS, for the ESSnet DI on-the-job training course, Southampton, 25-28 January 2011
-
Deduplication dataset taken from RecordLinkage R package developed by Murat Sariyar and Andreas Borg. Package is licensed under GPL-3 license. Also known as RLdata10000.
The files are hosted on GitHub Releases and can be downloaded via the provided links.
BlockingPy is released under MIT license.
BlockingPy benefits from many open-source packages such as Faiss or Annoy. For detailed information see third party notice.
Please see CONTRIBUTING.md for more information.
You can find it here
This package is based on the R blocking package developed by BERENZ.
Work on this package is supported by the National Science Centre, OPUS 20 grant no. 2020/39/B/HS4/00941 (Towards census-like statistics for foreign-born populations -- quality, data integration and estimation)