Commit graph

229 commits

Author SHA1 Message Date
karlicoss
c63e80ce94 core: more consistent handling of zip archives in get_files + tests 2023-10-20 21:13:59 +01:00
Dima Gerasimov
29832a9f75 core: fix test_get_files after updating kompress 2023-10-19 02:26:28 +01:00
karlicoss
fe26efaea8 core/kompress: move vendorized to _deprecated, use kompress library directly 2023-10-12 23:47:05 +01:00
karlicoss
bb478f369d core/logging: no need for super call in Filter 2023-10-12 23:47:05 +01:00
karlicoss
68289c1be3 general: fix ignores after mypy version update 2023-10-12 23:47:05 +01:00
Dima Gerasimov
0512488241 ci: sync configs to pymplate
- add python3.12
- add ruff
2023-10-06 02:24:01 +01:00
Dima Gerasimov
01480ec8eb core/logging: fix issue with logger setup called multiple times when called with different levels
should resolve https://github.com/karlicoss/HPI/issues/308
2023-09-19 22:39:52 +01:00
Sean Breckenridge
2a46341ce2 my.core.logging: compatibility with HPI_LOGS
re-adds a removed check for HPI_LOGS, add some docs

fix the checks for browserexport/takeout logs to
use the computed level from my.core.logging
2023-09-07 02:36:26 +01:00
Dima Gerasimov
c283e542e3 general: fix some issues after mypy update 2023-08-24 23:46:23 +01:00
Sean Breckenridge
fcaa7c1561 core/cli: allow user to bypass PEP 668
when installing dependencies with 'hpi module install',
this now lets a user pass '--break-system-packages' (or '-B'),
which passes the same option down to pip, to allow the user
to bypass PEP 668 and install packages that could possibly
conflict with system packages.
2023-08-10 01:41:43 +01:00
Dima Gerasimov
c25ab51664 core: some tweaks for better colour handling when we're redirecting stdout/stderr 2023-06-21 20:42:10 +01:00
Dima Gerasimov
661714f1d9 core/logging: overhaul and many improvements -- mainly to deprecate abandoned logzero
- generally saner/cleaner logger initialization

  In particular now it doesn't override logging level specified by the user code prior to instantiating the logger.

  Also remove the `LazyLogger` hack, doesn't seem like it's necessary when the above is implemented.

- get rid of `logzero` which is archived and abandoned now, use `colorlog` for coloured logging formatter

- allow configuring log level via shell via `LOGGING_LEVEL_module_name=<level>`

  E.g. `LOGGING_LEVEL_rescuexport_dal=WARNING LOGGING_LEVEL_my_rescuetime=debug ./script.py`

- port `AddExceptionTraceback` from HPI/promnesia

- port `CollapseLogsHandler` from HPI/promnesia

  Also allow configuring from the shell, e.g. `LOGGING_COLLAPSE=<level>`

- add support for `enlighten` progress bar, so it can be shared between different projects

  See https://github.com/Rockhopper-Technologies/enlighten#readme

  This allows nice CLI progressbars, e.g. for parallel processing of different files from HPI:

    ghexport.dal[111]  29%|████████████████████████████████████████████████████████████████▏              |  29/100 [00:03<00:07, 10.03 files/s]
    rexport.dal[comments]  17%|████████████████████████████████████▋                                      | 115/682 [00:03<00:14, 39.15 files/s]
    my.instagram.android   0%|▎                                                                           |    3/2631 [00:02<34:50, 1.26 files/s]

  Currently off by default, and hidden behind an env variable (`ENLIGHTEN_ENABLE=true`)
2023-06-21 18:42:15 +01:00
Dima Gerasimov
6aa3d4225e sort out mypy after its update 2023-06-21 03:32:46 +01:00
Dima Gerasimov
ab7135d42f core: experimental import of my._init_hook to configure logging/warnings/env variables 2023-06-21 03:32:46 +01:00
Dima Gerasimov
c91534b966 set json files to empty dicts so they are at least valid jsons
(promnesia was stumbling over these, seems like the easiest fix :) )
2023-06-09 03:31:13 +01:00
Dima Gerasimov
5fe21240b4 core: move mcachew into my.core.cachew; use better typing annotations (copied from cachew) 2023-06-08 01:29:49 +01:00
Dima Gerasimov
f8cd31044e general: move reddit tests into my/tests + tweak my.core.cfg to be more reliable 2023-05-26 00:58:23 +01:00
Dima Gerasimov
9594caa1cd general: move most core tests inside my.core.tests package
- distributes tests alongside the package, might be convenient for package users
- removes some weird indirection (e.g. dummy test files improting tests from modules)
- makes the command line for tests cleaner (e.g. no need to remember to manually add files to tox.ini)
- tests automatically covered by mypy (so makes mypy runs cleaner and ultimately better coverage)

The (vague) convention is

- tests/somemodule.py -- testing my.core.somemodule, contains tests directly re
- tests/test_something.py -- testing a specific feature, e.g. test_get_files.py tests get_files methon only
2023-05-25 00:25:13 +01:00
Dima Gerasimov
04d976f937 my/core/pandas tests: fix weird pytest error when constructing dataclass inside a def
can quickly reproduce by running pytest tests/tz.py tests/core/test_pandas.py
possibly will be resolved after fix in pytest?
see https://github.com/pytest-dev/pytest/issues/7856
2023-05-24 22:32:44 +01:00
Dima Gerasimov
a98bc6daca my.core.pandas: rely on typing annotations from types-pandas 2023-05-24 22:32:44 +01:00
Dima Gerasimov
fe88380499 general: switch to using native 3.8 versions for cached_property/Literal/Protocol instead of compat 2023-05-16 01:18:30 +01:00
Dima Gerasimov
c34656e8fb general: update mypy config, seems that logs of type: ignore aren't necessary anymore 2023-05-16 01:18:30 +01:00
seanbreckenridge
7a32302d66
query: add --warn-exceptions, dateparser, docs (#290)
* query: add --warn-exceptions, dateparser, docs

added --warn-exceptions (like --raise-exceptions/--drop-exceptions, but
lets you pass a warn_func if you want to customize how the exceptions are
handled. By default this creates a logger in main and logs the exception

added dateparser as a fallback if its installed (it's not a strong dependency, but
I mentioned in the docs that it's useful for parsing dates/times)

added docs for query, and a few examples

--output gpx respects the --{drop,warn,raise}--exceptions flags, have
an example of that in the docs as well
2023-04-18 00:15:35 +01:00
seanbreckenridge
40de162fab
cli: add option to output locations to gpx files (#286)
* cli: add option to output locations to gpx files
2023-04-15 00:31:11 +01:00
Dima Gerasimov
d464b1e607 core: implement more methods for ZipPath and better support for get_files 2023-04-03 22:58:54 +01:00
Kian-Meng Ang
d2ef23fcb4 docs: fix typos
found via `codespell -L copie,datas,pres,fo,tooks,noo,ue,ket,frop`
2023-03-27 03:02:35 +01:00
Dima Gerasimov
58d2e25a42 ci: suppress some mypy issues after upgrade 2023-03-17 00:33:22 +00:00
seanbreckenridge
79eeab2128
cli completion doc updates, hide legacy import warning (#279)
* core/cli: hide warnings when autocompleting

* link to completion in setup/troubleshooting
* update completion docs to make source path clear
2023-03-06 21:36:36 +00:00
Dima Gerasimov
a4c713664e core.logging: sync logging helper with Promnesia, adds more goodies
- print exception traceback by default when using logger.exception
- COLLAPSE_DEBUG_LOGS env variable
2023-03-03 21:14:11 +00:00
Dima Gerasimov
4dfc4029c3 core.kompress: proper support for read_text/read_bytes against zstd/xz archives 2023-03-03 21:14:11 +00:00
Dima Gerasimov
b94904f5ee core.kompress: support .zst extension, seems more conventional than .zstd 2023-03-03 21:14:11 +00:00
Sean Breckenridge
db2cd00bed try removing parallel on mac to prevent CI failure 2023-02-28 20:55:12 +00:00
seanbreckenridge
98b086f746
location fallback (#263)
see https://github.com/karlicoss/HPI/issues/262

* move home to fallback/via_home.py
* move via_ip to fallback
* add fallback model
* add stub via_ip file
* add fallback_locations for via_ip
* use protocol for locations
* estimate_from helper, via_home estimator, all.py
* via_home: add accuracy, cache history
* add datasources to gpslogger/google_takeout
* tz/via_location.py: update import to fallback
* denylist docs/installation instructions
* tz.via_location: let user customize cachew refresh time
* add via_ip.estimate_location using binary search
* use estimate_location in via_home.get_location
* tests: add gpslogger to location config stub
* tests: install tz related libs in test env
* tz: add regression test for broken windows dates

* vendorize bisect_left from python src
doesnt have a 'key' parameter till python3.10
2023-02-28 04:30:06 +00:00
Dima Gerasimov
6493859ba5 my.telegram: initial module from telegram_backup 2023-02-19 01:20:38 +00:00
Dima Gerasimov
0e884fe166 core/modules: switch away from using override_config to tmp_config in some tests & faka data generators 2023-02-09 02:35:09 +00:00
Dima Gerasimov
5ac5636e7f core: better support for ad-hoc configs
properly reload/unload the relevant modules so hopefully no more weird hacks should be required

relevant
- https://github.com/karlicoss/promnesia/issues/340
- https://github.com/karlicoss/HPI/issues/46
2023-02-09 02:35:09 +00:00
Dima Gerasimov
bb5ad2b6ac core: make hpi install more defensive, just warn on no requirements
this is useful for backwards compatibility if modules remove their requirements
2023-02-07 01:57:00 +00:00
Dima Gerasimov
5c82d0faa9 switch from using dataset to raw sqlite3 module
dataset is kinda unmaintaned and currently broken due to sqlalchemy 2.0 changes

resolves https://github.com/karlicoss/HPI/issues/264
2023-02-07 01:57:00 +00:00
Sean Breckenridge
54e6fe6ab5 ci: try disabling parallel pip installs on windows 2022-12-17 21:07:30 +00:00
Sean Breckenridge
716a2c82ba core/serialize: serialize stdlib Decimal class 2022-10-19 00:07:30 +01:00
Dima Gerasimov
7098d6831f fix mypy in _identity
seems easier to just ignore considering it's "internal" function

also a couple of tests to make sure it infers types correctly
2022-10-19 00:06:23 +01:00
Dima Gerasimov
119b295d71 core: allow legacy modules to be used in 'hpi module install' for backwards compatibility
but show warning

kinda hacky, but hopefully we will simplify it further when we have more such legacy modules
2022-06-07 22:59:08 +01:00
Sean Breckenridge
dbd15a7ee8 source: propogate help url for config errors 2022-06-07 21:33:38 +01:00
Dima Gerasimov
f0397b00ff core/main: experimental --parallel flag for hpi module install 2022-06-06 09:49:15 +01:00
Dima Gerasimov
5f0231c5ee core/main: allow passing multiple packages to 'module install'/'module requires' subcommands 2022-06-06 09:49:15 +01:00
Dima Gerasimov
016f28250b general: initial flake8 checks (for now manual)
fix fairly uncontroversial stuff in my.core like
- line spacing, which isn't too annoying (e.g. unlike many inline whitespace checks that break vertical formatting)
- unused imports/variables
- too broad except
2022-06-05 22:28:38 +01:00
Dima Gerasimov
bf3dd6e931 core/sqlite: experiment at typing SELECT query (to some extent)
ideally would be cool to use TypedDict here somehow, but perhaps it'd only be possible after variadic generics https://peps.python.org/pep-0646
2022-06-03 23:49:27 +01:00
Dima Gerasimov
7a1b7b1554 core/general: add assert_never + typing annotations for dataset 2022-06-03 23:49:27 +01:00
Dima Gerasimov
3faebdd629 core: add Protocol/TypedDict to compat 2022-06-02 14:21:21 +01:00
Dima Gerasimov
186f561018 core: some cleanup for core/init and doctor; fix issue with compileall 2022-06-02 14:21:21 +01:00