Metadata-Version: 2.1
Name: semhash
Version: 0.1.0
Summary: Fast Semantic Text Deduplication
Author-email: Thomas van Dongen <thomas123@live.nl>, Stéphan Tulkens <stephantul@gmail.com>
License: MIT License
        
        Copyright (c) 2024 The Minish Lab
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/MinishLab
Project-URL: Bug Reports, https://github.com/MinishLab/semhash/issues
Project-URL: Source, https://github.com/MinishLab/semhash
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Natural Language :: English
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: model2vec>=0.3.0
Requires-Dist: vicinity[usearch]>=0.3.4
Provides-Extra: dev
Requires-Dist: black; extra == "dev"
Requires-Dist: ipython; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-coverage; extra == "dev"
Requires-Dist: ruff; extra == "dev"


<div align="center">

# SemHash: Fast Semantic Text Deduplication


[Quickstart](#quickstart) •
[Main Features](#main-features) •
[Usage](#usage)
[Benchmarks](#benchmarks)

</div>


SemHash is a technique to efficiently deduplicate datasets based on semantic similarity. It uses a combination of lightning-fast embeddings through [model2vec](https://github.com/MinishLab/model2vec) and ANN-based similarity search through [vicinity](https://github.com/MinishLab/vicinity).



## Quickstart

Install the package with:
```bash
pip install semhash
```

Deduplicate a single dataset with the following code:

```python
from model2vec import StaticModel
from semhash import SemHash

# Load an embedding model
model = StaticModel.from_pretrained("minishlab/potion-base-8M")

# Initialize a SemHash with the model
semhash = SemHash(model=model)

# Create some texts to deduplicate
texts = [
        "It's dangerous to go alone!",
        "It's dangerous to go alone!",  # Exact duplicate
        "It's not safe to go by yourself!",  # Semantically similar
]

# Deduplicate the texts
deduplicated_texts = semhash.fit_deduplicate(records=texts, threshold=0.5)
```

For more advanced usage, you can also deduplicate across multiple datasets, or deduplicate multi-column datasets. Examples are provided in the [usage](#usage) section.


## Main Features

- **Fast**: SemHash uses model2vec to embed texts and vicinity to perform similarity search, making it extremely fast.
- **Scalable**: SemHash can deduplicate large datasets with millions of records thanks to the ANN backends in Vicinity.
- **Flexible**: SemHash can be used to deduplicate a single dataset or across two datasets, and can also be used to deduplicate multi-column datasets (such as QA datasets).
- **Lightweight**: SemHash is a lightweight package with minimal dependencies, making it easy to install and use.

## Usage

<details>
<summary>  Deduplicate a single dataset </summary>
<br>

The following code snippet shows how to deduplicate a single dataset using SemHash:

```python
from model2vec import StaticModel
from semhash import SemHash

# Load an embedding model
model = StaticModel.from_pretrained("minishlab/potion-base-8M")

# Initialize a SemHash with the model
semhash = SemHash(model=model)

# Create some texts to deduplicate
texts = [
        "It's dangerous to go alone!",
        "It's dangerous to go alone!",  # Exact duplicate
        "It's not safe to go by yourself!",  # Semantically similar
]

# Deduplicate the texts
deduplicated_texts = semhash.fit_deduplicate(records=texts, threshold=0.5)
```
</details>

<details>
<summary>  Deduplicate across two datasets </summary>
<br>

The following code snippet shows how to deduplicate across two datasets using SemHash (in this example, a training and test dataset):

```python
from model2vec import StaticModel
from semhash import SemHash

# Load an embedding model
model = StaticModel.from_pretrained("minishlab/potion-base-8M")

# Initialize a SemHash with the model
semhash = SemHash(model=model)

# Create some texts to deduplicate
train = [
    "It's dangerous to go alone!",
    "It's a secret to everybody.",
    "Ganondorf has invaded Hyrule!",
]
test = [
    "It's dangerous to go alone!",  # Exact duplicate
    "It's not safe to go by yourself!",  # Semantically similar
    "The master sword seals the darkness",
]

# Fit on the training data
semhash.fit(records=train)
# Deduplicate the test data against the training data
deduplicated_texts = semhash.deduplicate(records=test, threshold=0.5)
```

</details>

<details>
<summary>  Deduplicate multi-column datasets </summary>
<br>

The following code snippet shows how to deduplicate multi-column datasets using SemHash (in this example, a QA dataset with questions, contexts, and answers):

```python
from model2vec import StaticModel
from semhash import SemHash

# Load an embedding model
model = StaticModel.from_pretrained("minishlab/potion-base-8M")

# Initialize a SemHash with the model and columns to deduplicate
semhash = SemHash(model=model, columns=["question", "context", "answer"])

# Create some texts to deduplicate
records = [
    {"question": "What is the hero's name?", "context": "The hero is Link", "answer": "Link"},
    {"question": "What is the hero's name?", "context": "The hero is Link", "answer": "Link"},  # Exact duplicate
    {
        "question": "Who is the protagonist?",
        "context": "In this story, Link is the hero",
        "answer": "Link",
    },  # Semantically similar
    {"question": "Who is the princess?", "context": "The princess is Zelda", "answer": "Zelda"},
]

# Deduplicate the records
deduplicated_records = semhash.fit_deduplicate(records=records, threshold=0.5)
```

</details>

## Benchmarks

We've benchmarked SemHash on a variety of datasets to measure the deduplication performance and speed. The benchmarks were run with the following setup:
- The benchmarks were all run on CPU
- The benchmarks were all run with `ann=True`
- The timings include the encoding time, index building time, and deduplication time.

### Train Deduplication Benchmark

| Dataset | Original Train Size | Deduplicated Train Size | % Removed | Deduplication Time (s) |
| --- | --- | --- | --- | --- |
| bbc | 1225 | 1144 | 6.61 | 0.23 |
| senteval_cr | 3012 | 2990 | 0.73 | 0.13 |
| tweet_sentiment_extraction | 27481 | 26695 | 2.86 | 1.64 |
| emotion | 16000 | 15696 | 1.90 | 0.66 |
| amazon_counterfactual | 5000 | 4992 | 0.16 | 0.31 |
| ag_news | 120000 | 106921 | 10.90 | 4.21 |
| enron_spam | 31716 | 20539 | 35.24 | 1.57 |
| subj | 8000 | 7990 | 0.12 | 0.57 |
| sst5 | 8544 | 8526 | 0.21 | 0.55 |
| 20_newgroups | 11314 | 10685 | 5.56 | 0.69 |
| hatespeech_offensive | 22783 | 22090 | 3.04 | 0.84 |
| ade | 17637 | 15718 | 10.88 | 0.68 |
| imdb | 25000 | 24832 | 0.67 | 1.65 |
| massive_scenario | 11514 | 9366 | 18.66 | 0.43 |
| student | 117519 | 63865 | 45.66 | 4.18 |
| squad_v2 | 130319 | 115546 | 11.34 | 11.20 |
| wikitext | 1801350 | 884586 | 50.89 | 56.11 |


### Train/Test Deduplication Benchmark

| Dataset | Train Size | Test Size | Deduplicated Test Size | % Removed | Deduplication Time (s) |
| --- | --- | --- | --- | --- | --- |
| bbc | 1225 | 1000 | 875 | 12.50 | 0.39 |
| senteval_cr | 3012 | 753 | 750 | 0.40 | 0.11 |
| tweet_sentiment_extraction | 27481 | 3534 | 3411 | 3.48 | 0.86 |
| emotion | 16000 | 2000 | 1926 | 3.70 | 0.56 |
| amazon_counterfactual | 5000 | 5000 | 4990 | 0.20 | 0.48 |
| ag_news | 120000 | 7600 | 6197 | 18.46 | 3.38 |
| enron_spam | 31716 | 2000 | 1063 | 46.85 | 1.98 |
| subj | 8000 | 2000 | 1999 | 0.05 | 0.58 |
| sst5 | 8544 | 2210 | 2205 | 0.23 | 0.56 |
| 20_newgroups | 11314 | 7532 | 7310 | 2.95 | 2.26 |
| hatespeech_offensive | 22783 | 2000 | 1926 | 3.70 | 0.72 |
| ade | 17637 | 5879 | 4954 | 15.73 | 0.82 |
| imdb | 25000 | 25000 | 24805 | 0.78 | 2.65 |
| massive_scenario | 11514 | 2974 | 2188 | 26.43 | 0.43 |
| student | 117519 | 5000 | 2395 | 52.10 | 3.02 |
| squad_v2 | 130319 | 11873 | 11869 | 0.03 | 9.11 |
| wikitext | 1801350 | 4358 | 3610 | 17.16 | 36.10 |


As can be seen, SemHash is extremely fast, and scales to large datasets with millions of records. There are some notable examples of train/test leakage, such as `enron_spam` and `student`, where the test dataset contains a significant amount of semantic overlap with the training dataset.

### Reproducing the Benchmarks

To run the benchmarks yourself, you can use the following command:

```bash
python -m benchmarks.run_benchmarks
```
Optionally, the datasets can be updated in the [datasets.py](https://github.com/MinishLab/semhash/blob/main/benchmarks/datasets.py) file.
