fz0x1/HPI
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1053 commits 3 branches 41 tags 2.5 MiB
Commit graph

30 commits

Author SHA1 Message Date
karlicoss
71cb66df5f core: add helper for more_iterable to check that all types involved are hashable 
Otherwise unique_everseen performance may degrade to quadratic rather than linear

For now hidden behind HPI_CHECK_UNIQUE_EVERSEEN flag

also switch some modules to use it
 2023-10-31 01:02:17 +00:00
karlicoss
79ce8e84ec fbmessenger.android: support processing msys database 
seems that threads_db2 stopped updating some time ago, and msys contains all new data now
 2023-10-30 02:54:22 +00:00
karlicoss
f28f68b14b general: enhancle logging for various modules 2023-10-29 22:32:07 +00:00
Dima Gerasimov
4f7c9b4a71 core: move split compat/legacy modules into hpi_compat and compat 2023-10-28 20:06:54 +01:00
Dima Gerasimov
0512488241 ci: sync configs to pymplate 
- add python3.12
- add ruff
 2023-10-06 02:24:01 +01:00
Dima Gerasimov
fe88380499 general: switch to using native 3.8 versions for cached_property/Literal/Protocol instead of compat 2023-05-16 01:18:30 +01:00
Kian-Meng Ang
d2ef23fcb4 docs: fix typos 
found via `codespell -L copie,datas,pres,fo,tooks,noo,ue,ket,frop`
 2023-03-27 03:02:35 +01:00
Dima Gerasimov
347cd1ef77 my.fbmessenger: add Sender protocol for consistency 2023-03-17 00:33:22 +00:00
Dima Gerasimov
bef832cbff my.fbmessenger.export: remove legacy dump_chat_history code 2023-03-17 00:33:22 +00:00
Dima Gerasimov
0a05b27266 my.fbmessenger.android: set timezone to utc 2023-03-17 00:33:22 +00:00
Dima Gerasimov
bee17d932b fbmessenger.android: use Optional name, best to leave for the consumer to decide how to behave when it's unavailable 
e.g. using  <NAME UNAVAILABLE> was causing issues when used as zulip contact name
 2023-03-03 21:14:11 +00:00
Dima Gerasimov
c63177e186 general/ci: clean up mypy-misc pipeline, only exclude specific files instead 
marked some module configs which aren't really ready for public use as type: ignore
 2023-02-21 00:20:58 +00:00
Dima Gerasimov
eff9c02886 my.fbmessenger.android: add optional facebook_id 2023-02-20 20:14:15 +00:00
Dima Gerasimov
af874d2d75 my.fbmessenger.android: minor refactoring, comments & error handling 2023-02-20 20:14:15 +00:00
Dima Gerasimov
fb0c1289f0 my.fbmessenger.export: use context manager to properly close sqlite connection 2023-02-08 02:18:00 +00:00
Dima Gerasimov
5c82d0faa9 switch from using dataset to raw sqlite3 module 
dataset is kinda unmaintaned and currently broken due to sqlalchemy 2.0 changes

resolves https://github.com/karlicoss/HPI/issues/264
 2023-02-07 01:57:00 +00:00
Dima Gerasimov
b9d788efd0 some enhancements for facebook/instagram modules 
figured out that datetimes are naive
better username handling + investigation of thread names
 2022-06-04 17:16:28 +01:00
Dima Gerasimov
b5f266c2bd my.instagram: add initial all.py + some experiments on nicer errors 2022-06-03 23:49:27 +01:00
Dima Gerasimov
7a1b7b1554 core/general: add assert_never + typing annotations for dataset 2022-06-03 23:49:27 +01:00
Dima Gerasimov
b96c9f4534 fbmessenger: use both id and timestamp for merging 2022-06-02 14:21:21 +01:00
Dima Gerasimov
3faebdd629 core: add Protocol/TypedDict to compat 2022-06-02 14:21:21 +01:00
Dima Gerasimov
9461df6aa5 general: extract the hack to warn of legacy imports and fallback to core/legacy.py 
use it both in my.fbmessenger and my.reddit

if in the future any new modules need to be switched to namespace package structure with all.py it should make it easy to do

related:
- https://github.com/karlicoss/HPI/issues/12
- https://github.com/karlicoss/HPI/issues/89
- https://github.com/karlicoss/HPI/issues/102
 2022-06-01 23:27:34 +01:00
Dima Gerasimov
8336d18434 general: add an adhoc test for checking mixin behaviour with namespace packages and __init__.py hack 
also use that hack in my.fbmessenger
 2022-06-01 23:27:34 +01:00
Dima Gerasimov
2025d7ad1a general: minor cleanup 
- get rid of unnecessary globs in get_files (they should be in config if the user wishes)
- get rid of some old kython imports
- do not convert Path twice in foursquare (so CPath works correctly)
 2022-05-31 22:16:05 +01:00
Dima Gerasimov
b9852f45cf twitter: use import_source and proper merging for tweets from different sources 
+ use proper datetime_aware for created_at
 2022-02-08 20:45:10 +00:00
Dima Gerasimov
f8e73134b3 fbmessenger: add all.py, merge messages from different sources 
followup for https://github.com/karlicoss/HPI/pull/179
 2022-02-08 19:21:44 +00:00
Dima Gerasimov
4626c1bba6 fbmessenger: support config migration for fbmessengerexport source 
for now kinda copied from reddit... still thinking about a more generic way
 2022-02-05 14:49:12 +00:00
Dima Gerasimov
403ca9c111 fbmessenger: process Android app data 
for now, no merging, will figure it out later
 2022-02-05 14:49:12 +00:00
Dima Gerasimov
fcd7ca6480 fbmessenger: only import from .export in legacy mode 2022-02-05 14:49:12 +00:00
Dima Gerasimov
d1f791dee8 my.fbmessenger: move fbmessenger.py into fbmessenger/export.py 
keeping it backwards compatible + conditional warning similar to https://github.com/karlicoss/HPI/pull/179

follow up for https://github.com/seanbreckenridge/HPI/pull/18
for now without the __path__ hacking, will do it in bulk later

too lazy to run test_import_warnings.sh on CI for now, but figured I'd commit it for the reference anyway
 2022-02-02 23:22:45 +00:00