this speeds up processing quite a bit, from 40s to 20s for me, plus removes tons of identical outputs
interesting enough, using raw object without json.dumps as key brings unique_everseen to crawl...
when installing dependencies with 'hpi module install',
this now lets a user pass '--break-system-packages' (or '-B'),
which passes the same option down to pip, to allow the user
to bypass PEP 668 and install packages that could possibly
conflict with system packages.
- generally saner/cleaner logger initialization
In particular now it doesn't override logging level specified by the user code prior to instantiating the logger.
Also remove the `LazyLogger` hack, doesn't seem like it's necessary when the above is implemented.
- get rid of `logzero` which is archived and abandoned now, use `colorlog` for coloured logging formatter
- allow configuring log level via shell via `LOGGING_LEVEL_module_name=<level>`
E.g. `LOGGING_LEVEL_rescuexport_dal=WARNING LOGGING_LEVEL_my_rescuetime=debug ./script.py`
- port `AddExceptionTraceback` from HPI/promnesia
- port `CollapseLogsHandler` from HPI/promnesia
Also allow configuring from the shell, e.g. `LOGGING_COLLAPSE=<level>`
- add support for `enlighten` progress bar, so it can be shared between different projects
See https://github.com/Rockhopper-Technologies/enlighten#readme
This allows nice CLI progressbars, e.g. for parallel processing of different files from HPI:
ghexport.dal[111] 29%|████████████████████████████████████████████████████████████████▏ | 29/100 [00:03<00:07, 10.03 files/s]
rexport.dal[comments] 17%|████████████████████████████████████▋ | 115/682 [00:03<00:14, 39.15 files/s]
my.instagram.android 0%|▎ | 3/2631 [00:02<34:50, 1.26 files/s]
Currently off by default, and hidden behind an env variable (`ENLIGHTEN_ENABLE=true`)
- distributes tests alongside the package, might be convenient for package users
- removes some weird indirection (e.g. dummy test files improting tests from modules)
- makes the command line for tests cleaner (e.g. no need to remember to manually add files to tox.ini)
- tests automatically covered by mypy (so makes mypy runs cleaner and ultimately better coverage)
The (vague) convention is
- tests/somemodule.py -- testing my.core.somemodule, contains tests directly re
- tests/test_something.py -- testing a specific feature, e.g. test_get_files.py tests get_files methon only
