fz0x1/HPI
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

core: add helper for more_iterable to check that all types involved are hashable

Browse source 
Otherwise unique_everseen performance may degrade to quadratic rather than linear

For now hidden behind HPI_CHECK_UNIQUE_EVERSEEN flag

also switch some modules to use it
This commit is contained in:
karlicoss 2023-10-31 00:42:17 +00:00
parent d6786084ca
commit 71cb66df5f
8 changed files with 90 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

8
my/vk/vk_messages_backup.py
View file

@ -5,12 +5,12 @@ VK data (exported by [[https://github.com/Totktonada/vk_messages_backup][Totkton
from datetime import datetime
from dataclasses import dataclass
import json
from typing import Dict, Iterator, NamedTuple
from typing import Dict, Iterator
from more_itertools import unique_everseen
import pytz
from my.core import stat, Stats, Json, Res, datetime_aware
from my.core import stat, Stats, Json, Res, datetime_aware, get_files
from my.core.common import unique_everseen
from my.config import vk_messages_backup as config
@ -147,7 +147,7 @@ def _messages() -> Iterator[Res[Message]]:
def messages() -> Iterator[Res[Message]]:
# seems that during backup messages were sometimes duplicated..
yield from unique_everseen(_messages())
yield from unique_everseen(_messages)
def stats() -> Stats: