Commit graph

972 commits

Author SHA1 Message Date
Dima Gerasimov
c91534b966 set json files to empty dicts so they are at least valid jsons
(promnesia was stumbling over these, seems like the easiest fix :) )
2023-06-09 03:31:13 +01:00
Dima Gerasimov
5fe21240b4 core: move mcachew into my.core.cachew; use better typing annotations (copied from cachew) 2023-06-08 01:29:49 +01:00
Dima Gerasimov
f8cd31044e general: move reddit tests into my/tests + tweak my.core.cfg to be more reliable 2023-05-26 00:58:23 +01:00
Dima Gerasimov
fcfc423a75 move some tests into the main HPI package 2023-05-26 00:03:24 +01:00
Dima Gerasimov
9594caa1cd general: move most core tests inside my.core.tests package
- distributes tests alongside the package, might be convenient for package users
- removes some weird indirection (e.g. dummy test files improting tests from modules)
- makes the command line for tests cleaner (e.g. no need to remember to manually add files to tox.ini)
- tests automatically covered by mypy (so makes mypy runs cleaner and ultimately better coverage)

The (vague) convention is

- tests/somemodule.py -- testing my.core.somemodule, contains tests directly re
- tests/test_something.py -- testing a specific feature, e.g. test_get_files.py tests get_files methon only
2023-05-25 00:25:13 +01:00
Dima Gerasimov
04d976f937 my/core/pandas tests: fix weird pytest error when constructing dataclass inside a def
can quickly reproduce by running pytest tests/tz.py tests/core/test_pandas.py
possibly will be resolved after fix in pytest?
see https://github.com/pytest-dev/pytest/issues/7856
2023-05-24 22:32:44 +01:00
Dima Gerasimov
a98bc6daca my.core.pandas: rely on typing annotations from types-pandas 2023-05-24 22:32:44 +01:00
Dima Gerasimov
fe88380499 general: switch to using native 3.8 versions for cached_property/Literal/Protocol instead of compat 2023-05-16 01:18:30 +01:00
Dima Gerasimov
c34656e8fb general: update mypy config, seems that logs of type: ignore aren't necessary anymore 2023-05-16 01:18:30 +01:00
Dima Gerasimov
a445d2cbfe general: python3.7 will reach EOL soon, remove its support 2023-05-16 01:18:30 +01:00
seanbreckenridge
7a32302d66
query: add --warn-exceptions, dateparser, docs (#290)
* query: add --warn-exceptions, dateparser, docs

added --warn-exceptions (like --raise-exceptions/--drop-exceptions, but
lets you pass a warn_func if you want to customize how the exceptions are
handled. By default this creates a logger in main and logs the exception

added dateparser as a fallback if its installed (it's not a strong dependency, but
I mentioned in the docs that it's useful for parsing dates/times)

added docs for query, and a few examples

--output gpx respects the --{drop,warn,raise}--exceptions flags, have
an example of that in the docs as well
2023-04-18 00:15:35 +01:00
Sean Breckenridge
82bc51d9fc smscalls: make checking for keys stricter
sort of reverts #287, but also makes some other improvements

this allows us to remove some of the Optional's to
make downstream consumers easier to write. However,
this keeps the return type as a Res (result, with errors),
so downstream consumers will have to handle those incase
the schema ever changes (highly unlikely)

also added the 'call_type/message_type' with a comment
there describing the values

I left 'who' Optional I believe it actually should be -
its very possible for there to be no contact name, added
a check incase its '(Unknown)' which is what my phone
sets it to
2023-04-15 17:17:02 +01:00
seanbreckenridge
40de162fab
cli: add option to output locations to gpx files (#286)
* cli: add option to output locations to gpx files
2023-04-15 00:31:11 +01:00
Sean Breckenridge
02c738594f smscalls: make some fields optional, yield errors
reflects the new types-lxml package
https://github.com/abelcheung/types-lxml
2023-04-14 23:50:26 +01:00
Dima Gerasimov
d464b1e607 core: implement more methods for ZipPath and better support for get_files 2023-04-03 22:58:54 +01:00
Dima Gerasimov
0c5b2b4a09 my.whatsapp.android: initial module 2023-04-01 04:07:35 +01:00
Dima Gerasimov
8288032b1c my.telegram.telegram_backup: support optional extra_where and optional media info extraction for Promnesia 2023-03-27 03:27:13 +01:00
Dima Gerasimov
74710b339a telegram_backup: order messages by date and users/chats by id for determinism 2023-03-27 03:27:13 +01:00
Kian-Meng Ang
d2ef23fcb4 docs: fix typos
found via `codespell -L copie,datas,pres,fo,tooks,noo,ue,ket,frop`
2023-03-27 03:02:35 +01:00
Dima Gerasimov
919c84fb5a my.instagram: better unification of like messages/reactions 2023-03-27 02:16:17 +01:00
Dima Gerasimov
9aadbb504b my.instagram.android: properly extract our own user 2023-03-27 02:16:17 +01:00
Dima Gerasimov
8f7d14e7c6 my.instagram: somewhat mad merging mechanism to correlate gdpr and android exports 2023-03-27 02:16:17 +01:00
Dima Gerasimov
e7be680841 my.instagram.gdpr: handle missing message content defensively 2023-03-27 02:16:17 +01:00
Dima Gerasimov
347cd1ef77 my.fbmessenger: add Sender protocol for consistency 2023-03-17 00:33:22 +00:00
Dima Gerasimov
58d2e25a42 ci: suppress some mypy issues after upgrade 2023-03-17 00:33:22 +00:00
Dima Gerasimov
bef832cbff my.fbmessenger.export: remove legacy dump_chat_history code 2023-03-17 00:33:22 +00:00
Dima Gerasimov
0a05b27266 my.fbmessenger.android: set timezone to utc 2023-03-17 00:33:22 +00:00
Dima Gerasimov
457797bdfb my.bumble.android: better handling for missing conversation id in database 2023-03-17 00:33:22 +00:00
Dima Gerasimov
9db5f318fb my.twitter.twint: use dict row factory instead of sqlite Row
otherwise it's not json serializable
2023-03-17 00:33:22 +00:00
seanbreckenridge
79eeab2128
cli completion doc updates, hide legacy import warning (#279)
* core/cli: hide warnings when autocompleting

* link to completion in setup/troubleshooting
* update completion docs to make source path clear
2023-03-06 21:36:36 +00:00
seanbreckenridge
9d231a8ea9
google_takeout: add semantic location history (#278)
* google_takeout: add semantic location history
2023-03-04 18:36:10 +00:00
Dima Gerasimov
a4c713664e core.logging: sync logging helper with Promnesia, adds more goodies
- print exception traceback by default when using logger.exception
- COLLAPSE_DEBUG_LOGS env variable
2023-03-03 21:14:11 +00:00
Dima Gerasimov
bee17d932b fbmessenger.android: use Optional name, best to leave for the consumer to decide how to behave when it's unavailable
e.g. using  <NAME UNAVAILABLE> was causing issues when used as zulip contact name
2023-03-03 21:14:11 +00:00
Dima Gerasimov
4dfc4029c3 core.kompress: proper support for read_text/read_bytes against zstd/xz archives 2023-03-03 21:14:11 +00:00
Dima Gerasimov
b94904f5ee core.kompress: support .zst extension, seems more conventional than .zstd 2023-03-03 21:14:11 +00:00
Sean Breckenridge
db2cd00bed try removing parallel on mac to prevent CI failure 2023-02-28 20:55:12 +00:00
Sean Breckenridge
a70118645b my.ip.common: remove REQUIRES
no reason to have it there since its
__NOT_HPI_MODULE__, so is not discoverable anyways
2023-02-28 20:55:12 +00:00
Sean Breckenridge
f36bc6144b tox: use my.ip.all, sort hpi installs 2023-02-28 20:55:12 +00:00
Sean Breckenridge
435cb020f9 add example for denylist, update ci 2023-02-28 20:55:12 +00:00
seanbreckenridge
98b086f746
location fallback (#263)
see https://github.com/karlicoss/HPI/issues/262

* move home to fallback/via_home.py
* move via_ip to fallback
* add fallback model
* add stub via_ip file
* add fallback_locations for via_ip
* use protocol for locations
* estimate_from helper, via_home estimator, all.py
* via_home: add accuracy, cache history
* add datasources to gpslogger/google_takeout
* tz/via_location.py: update import to fallback
* denylist docs/installation instructions
* tz.via_location: let user customize cachew refresh time
* add via_ip.estimate_location using binary search
* use estimate_location in via_home.get_location
* tests: add gpslogger to location config stub
* tests: install tz related libs in test env
* tz: add regression test for broken windows dates

* vendorize bisect_left from python src
doesnt have a 'key' parameter till python3.10
2023-02-28 04:30:06 +00:00
Dima Gerasimov
6dc5e7575f vk_messages_backup: add unique_everseen to prevent duplicate messages 2023-02-28 03:55:44 +00:00
Dima Gerasimov
a7099e2efc vk_messages_backup: more correct handling of group chats & better chat ids 2023-02-28 03:55:44 +00:00
Dima Gerasimov
02c98143d5 vk_messages_backup: better structure & exract richer information 2023-02-28 03:55:44 +00:00
Dima Gerasimov
130c273513 my.telegram.telegram_backup enhancements
- add chat handle
- add permalink
- more precise types
2023-02-21 02:44:18 +00:00
Dima Gerasimov
07e7c62d02 general/ci: mypy check tests 2023-02-21 00:20:58 +00:00
Dima Gerasimov
c63177e186 general/ci: clean up mypy-misc pipeline, only exclude specific files instead
marked some module configs which aren't really ready for public use as type: ignore
2023-02-21 00:20:58 +00:00
Dima Gerasimov
eff9c02886 my.fbmessenger.android: add optional facebook_id 2023-02-20 20:14:15 +00:00
Dima Gerasimov
af874d2d75 my.fbmessenger.android: minor refactoring, comments & error handling 2023-02-20 20:14:15 +00:00
Dima Gerasimov
6493859ba5 my.telegram: initial module from telegram_backup 2023-02-19 01:20:38 +00:00
Dima Gerasimov
6594ad24dc my.tinder.android: speedup unique_everseen by adding unsafe_hash 2023-02-19 01:20:38 +00:00