Commit graph

84 commits

Author SHA1 Message Date
Dima Gerasimov
adbc0e73a2 docs: add note about directly checking overlays with mypy 2023-12-22 02:54:36 +00:00
Dima Gerasimov
84d835962d docs: some documentation/thoughts on properly implementing overlay packages 2023-12-20 02:51:27 +00:00
karlicoss
bd27bd4c24 docs: add documentation on logging during HPI module development 2023-10-29 00:50:22 +01:00
Sean Breckenridge
2a46341ce2 my.core.logging: compatibility with HPI_LOGS
re-adds a removed check for HPI_LOGS, add some docs

fix the checks for browserexport/takeout logs to
use the computed level from my.core.logging
2023-09-07 02:36:26 +01:00
Dima Gerasimov
661714f1d9 core/logging: overhaul and many improvements -- mainly to deprecate abandoned logzero
- generally saner/cleaner logger initialization

  In particular now it doesn't override logging level specified by the user code prior to instantiating the logger.

  Also remove the `LazyLogger` hack, doesn't seem like it's necessary when the above is implemented.

- get rid of `logzero` which is archived and abandoned now, use `colorlog` for coloured logging formatter

- allow configuring log level via shell via `LOGGING_LEVEL_module_name=<level>`

  E.g. `LOGGING_LEVEL_rescuexport_dal=WARNING LOGGING_LEVEL_my_rescuetime=debug ./script.py`

- port `AddExceptionTraceback` from HPI/promnesia

- port `CollapseLogsHandler` from HPI/promnesia

  Also allow configuring from the shell, e.g. `LOGGING_COLLAPSE=<level>`

- add support for `enlighten` progress bar, so it can be shared between different projects

  See https://github.com/Rockhopper-Technologies/enlighten#readme

  This allows nice CLI progressbars, e.g. for parallel processing of different files from HPI:

    ghexport.dal[111]  29%|████████████████████████████████████████████████████████████████▏              |  29/100 [00:03<00:07, 10.03 files/s]
    rexport.dal[comments]  17%|████████████████████████████████████▋                                      | 115/682 [00:03<00:14, 39.15 files/s]
    my.instagram.android   0%|▎                                                                           |    3/2631 [00:02<34:50, 1.26 files/s]

  Currently off by default, and hidden behind an env variable (`ENLIGHTEN_ENABLE=true`)
2023-06-21 18:42:15 +01:00
seanbreckenridge
7a32302d66
query: add --warn-exceptions, dateparser, docs (#290)
* query: add --warn-exceptions, dateparser, docs

added --warn-exceptions (like --raise-exceptions/--drop-exceptions, but
lets you pass a warn_func if you want to customize how the exceptions are
handled. By default this creates a logger in main and logs the exception

added dateparser as a fallback if its installed (it's not a strong dependency, but
I mentioned in the docs that it's useful for parsing dates/times)

added docs for query, and a few examples

--output gpx respects the --{drop,warn,raise}--exceptions flags, have
an example of that in the docs as well
2023-04-18 00:15:35 +01:00
Kian-Meng Ang
d2ef23fcb4 docs: fix typos
found via `codespell -L copie,datas,pres,fo,tooks,noo,ue,ket,frop`
2023-03-27 03:02:35 +01:00
seanbreckenridge
79eeab2128
cli completion doc updates, hide legacy import warning (#279)
* core/cli: hide warnings when autocompleting

* link to completion in setup/troubleshooting
* update completion docs to make source path clear
2023-03-06 21:36:36 +00:00
Sean Breckenridge
435cb020f9 add example for denylist, update ci 2023-02-28 20:55:12 +00:00
seanbreckenridge
98b086f746
location fallback (#263)
see https://github.com/karlicoss/HPI/issues/262

* move home to fallback/via_home.py
* move via_ip to fallback
* add fallback model
* add stub via_ip file
* add fallback_locations for via_ip
* use protocol for locations
* estimate_from helper, via_home estimator, all.py
* via_home: add accuracy, cache history
* add datasources to gpslogger/google_takeout
* tz/via_location.py: update import to fallback
* denylist docs/installation instructions
* tz.via_location: let user customize cachew refresh time
* add via_ip.estimate_location using binary search
* use estimate_location in via_home.get_location
* tests: add gpslogger to location config stub
* tests: install tz related libs in test env
* tz: add regression test for broken windows dates

* vendorize bisect_left from python src
doesnt have a 'key' parameter till python3.10
2023-02-28 04:30:06 +00:00
Sean Breckenridge
7925ec81b6 docs: browser - fix examples for config 2022-08-29 00:03:32 +01:00
seanbreckenridge
0ce44bf0d1
doctor: better quick option propogation for stats (#239)
doctor: better quick option propogation for stats

* use contextmanager for quick stats instead of editing global state
  directly
* send quick to lots of stat related functions, so they
could possibly be used without doctor, if someone wanted to
* if a stats function has a 'quick' kwarg, send the value
there as well
* add an option to sort locations in my.time.tz.via_location
2022-05-02 00:13:05 +01:00
Sean Breckenridge
f43eedd52a docs: describe the all.py/import_source pattern 2022-04-27 07:57:16 +01:00
seanbreckenridge
2cb836181b
location: add all.py, using takeout/gpslogger/ip (#237)
* location: add all.py, using takeout/gpslogger/ip, update docs
2022-04-26 21:11:35 +01:00
Sean Breckenridge
66a00c6ada docs: add docs for google_takeout_parser 2022-04-25 02:52:34 +01:00
Sean Breckenridge
16c777b45a my.config: catch possible nested config errors 2022-04-10 16:51:15 +01:00
seanbreckenridge
9e5cd60ff2
browser: parse browser history using browserexport (#216)
* browser: parse browser history using browserexport

from seanbreckenridge/HPI module:
1fba8ccf2f/my/browser/export.py
2022-02-13 23:56:05 +00:00
Sean Breckenridge
059c4ae791 docs: add link to template 2022-02-11 09:33:03 +00:00
Sean Breckenridge
a791b25650 core/cli: add --debug flag, add HPI_LOGS to docs 2022-02-11 09:31:10 +00:00
Sean Breckenridge
3f4fb64d56
core: drop py36 support, update docs for reddit (#193)
* docs: update references to my.reddit
* ci: remove 3.6, add 3.9
2022-01-27 08:26:15 +00:00
Sean Breckenridge
8033b5cdbd docs/reddit: fix code block 2021-12-19 17:20:12 +00:00
Sean Breckenridge
4364484192 docs: add hpi query example, links to other repos
also updated the MODULE_DESIGN docs to mention the
current workaround for converting single file
modules to namespace packages through the
deprecation warning
2021-12-19 17:20:12 +00:00
Sean Breckenridge
8422c6e420
my.reddit: refactor into module that supports pushshift/gdpr (#179)
* initial pushshift/rexport merge implementation, using id for merging
* smarter module deprecation warning using regex
* add `RedditBase` from promnesia
* `import_source` helper for gracefully handing mixin data sources
2021-10-31 20:39:04 +00:00
Sean Breckenridge
4a04c09f31 docs: fix copy-paste errors/spelling mistakes 2021-07-10 10:56:23 +01:00
Dima Gerasimov
2a4bddea79 polar: move to top level, add page support 2021-05-29 13:26:01 +01:00
Sean Breckenridge
c1b70cd90e
docs: add some documentation on module design (#160) 2021-04-11 16:53:43 +01:00
Dima Gerasimov
ad1cc71b0f readme: update 2021-02-20 01:39:55 +00:00
Dima Gerasimov
56d5587c20 CI: clean up tox config a bit, get rid of custom lint script 2021-02-18 02:04:40 +00:00
Dima Gerasimov
f40b804833 update setup documentation 2020-11-01 02:02:43 +01:00
Dima Gerasimov
1849a66f08 general: get rid of example_config & use demo/stub my.config instead 2020-11-01 02:02:43 +01:00
Dima Gerasimov
3a9e3e080f my.time.tz: implement different policies for localizing 2020-11-01 01:51:10 +01:00
Dima Gerasimov
4666378f7e my.location.home: simplify config format, make it a bit more robust + tests 2020-10-12 09:05:11 +02:00
Dima Gerasimov
dfea664f57 add my.location.home, use it as location/timezone fallback 2020-10-09 22:09:19 +02:00
Dima Gerasimov
ba9acc3445 my.location: let takeout provider be in a separate my.location.google; add CI test & enable mypy 2020-10-08 21:31:26 +02:00
Dima Gerasimov
6242307d7a my.bluemaestro: run against testdata, add on CI 2020-10-04 01:40:52 +02:00
Dima Gerasimov
e63c159b80 my.body.exercise: add more annotations & ci check 2020-10-03 18:24:08 +02:00
Dima Gerasimov
d8841d0d7a my.endomondo: add fake data generator, test mypy 2020-10-02 00:37:08 +02:00
Dima Gerasimov
1c20eb27aa CI: add mypy checks for my.reddit, my.pocket and my.github.ghexport 2020-09-30 23:33:06 +02:00
Dima Gerasimov
0682919449 general: use module dependencies as proper PIP packages + fallback 2020-09-30 23:33:06 +02:00
Dima Gerasimov
dc642b5a6d my.instapaper: add stat; add mypy checks on CI 2020-09-29 20:43:34 +02:00
Dima Gerasimov
3404b3fcf1 my.instapaper: use instapexport from PIP package 2020-09-29 20:43:34 +02:00
Dima Gerasimov
24fb983399 ci: add mypy for my.hypothesis 2020-09-29 19:44:45 +02:00
Dima Gerasimov
deefa9fbbc Use hypexport package in demo.py, clean up tox 2020-09-29 19:44:45 +02:00
Adrien Lacquemant
5b2cc577f2 Correct command to create config file 2020-08-20 20:14:55 +01:00
Dima Gerasimov
821eb47c93 kobo: BREAKING changes. Use kobuddy module directly, rename export_dir to export_path.
Hopefully this makes a lot of sense in the first place, and not that many users, so deserves breaking.
2020-06-04 22:50:52 +01:00
Dima Gerasimov
3d7844b711 core: support '' for explicitly set empty path set 2020-06-01 23:45:26 +01:00
Dima Gerasimov
a267aeec5b github: add config templates + docs
- ghexport: use export_path (export_dir is still supported)
2020-06-01 23:33:34 +01:00
Dima Gerasimov
f175acc848 pocket: reuse pockexport data access layer
BREAKING CHANGE! Data parsing was switched to pockexport.
This would help to keep it consistent across different apps in the future.

When you update, you'll need to:

- clone pockexport (latest version)
- set pockexport repository in your config (see doc/MODULES.org)
2020-05-27 08:42:47 +01:00
Dima Gerasimov
6453ff415d docs: somewhat acceptable data flow diagrams 2020-05-26 22:51:50 +01:00
Dima Gerasimov
150a6a8cb7 docs: wip on better explanation of configs/diagram 2020-05-26 22:51:50 +01:00