Commit graph

138 commits

Author SHA1 Message Date
Dima Gerasimov
0517f7ffb8 core/influxdb: add main method to create influx measurement and fill with values
allows running something like

    python3 -m my.core.influxdb populate my.zotero
2021-04-27 20:10:10 +01:00
Sean Breckenridge
0278f2b68d cli/query: improve fallback behaviour/error msg 2021-04-24 06:15:59 +01:00
Dima Gerasimov
393ed0d9ce core: set _max_workers for dummy concurrent pool 2021-04-22 11:11:39 +01:00
Sean Breckenridge
4b4cb7cb5b cli/query: bugfix where datetime was ignored 2021-04-19 20:21:17 +01:00
Sean Breckenridge
277f0e3988
cli/query: interactive fallback, improve guess_stats (#163) 2021-04-19 18:57:42 +01:00
Dima Gerasimov
68d3385468 my.zotero: handle colors & extract human readable 2021-04-13 18:05:49 +01:00
Sean Breckenridge
fb49243005
core: add hpi query command (#157)
- restructure query code for cli, some test fixes
- initial query_range implementation

    refactored functions in query some more
    to allow re-use in range_range, select()
    pretty much just calls out to a bunch
    of handlers now
2021-04-06 17:19:58 +01:00
Dima Gerasimov
b94120deaf core/sqlite: add compat version for backup() for python3.6 2021-04-05 08:37:07 +01:00
Dima Gerasimov
f09ca17560 core/sqlite: move tests to separate module, pickling during Pool.submit can't handle importing :( 2021-04-05 08:37:07 +01:00
Dima Gerasimov
e99e8725b1 core/sqlite: add a helper to do an im-memory snapshot of db including WAL
+ add a bunch of tests for different WAL behaviours
2021-04-05 08:37:07 +01:00
Dima Gerasimov
f2a339f755 core/sqlite: extract immutable connection helper
use in bluemaestro/zotero modules
2021-04-05 08:37:07 +01:00
Sean Breckenridge
349ab78fca
core/cli: switch to using click library #155
everything is backwards-compatible with the previous
interface, the only minor changes were to the doctor cmd
which can now accept more than one item to run,
and the --skip-config-check to skip the config_ok
check if the user specifies to

added a test using click.testing.CliRunner (tests
the CLI in an isolated environment), though
additional tests which aren't testing the CLI
itself (parsing arguments or decorator behaviour)
can just call the functions themselves, as they
no longer accept a argparser.Namespace and instead
accept the direct arguments
2021-04-04 10:06:59 +01:00
Dima Gerasimov
b306ccc839 core: add ensure_unique iterator transfromation 2021-04-02 20:09:53 +01:00
Sean Breckenridge
c31641b4eb
core: discovery_pure; allow multiple package roots (#152)
* core: discovery_pure; allow multiple package roots

iterates over my.__path__._path if possible
to discover additional paths to iterate over

else defaults to the path relative to
the current file
2021-04-02 15:46:18 +01:00
Sean Breckenridge
5ecd4b4810 cleanup; remove unused imports 2021-04-02 08:38:06 +01:00
Dima Gerasimov
ad177a1ccd my.pdfs: cleanup/refactor
- modernize:
  - add REQUIRES spec for pdfannots library
  - config dataclass/config stub
  - stats function
  - absolute my.core imports in anticipation of splitting core
- use 'paths' instead of 'roots' (better reflects the semantics), use get_files
  backward compatible via config migration
- properly run tests/mypy
2021-04-01 17:27:06 +01:00
Dima Gerasimov
5c38872efc core: add DummyExecutor to make it easier to debug concurrent code with Pools 2021-04-01 17:27:06 +01:00
Sean Breckenridge
3118891c03
my.core.query: initial implementation (#143)
in particular `my.core.query.select`: a function to query, order, sort and filter items from one or more sources
2021-03-28 07:52:50 +01:00
Sean Breckenridge
d47f3c28aa my.core.serialize: support serializing Paths 2021-03-28 06:55:03 +01:00
Sean Breckenridge
1b36bd4379 fix spelling mistakes 2021-03-28 06:53:24 +01:00
Sean Breckenridge
1cdef6f40a fix mypy errors
this fixes two distinct mypy errors

one where NamedTuple/dataclassees can't be
defined locally
https://github.com/python/mypy/issues/7281

which happens when you run mypy like
mypy -p my.core on warm cache

the second error is the core/types.py file shadowing the
stdlib types module
2021-03-22 06:34:07 +00:00
Sean Breckenridge
eb26cf8633
my.core.serialize: orjson with additional default and _serialize hook (#140)
basic orjson serialize, json.dumps fallback

Lots of surrounding changes from this discussion:
0593c69056
2021-03-20 00:48:03 +00:00
Dima Gerasimov
8d6f691824 core: feature: guess module stats from typing annotations 2021-03-15 10:27:18 +00:00
Dima Gerasimov
1fd2a9f643 core/time: more flexible support for resolving TZ abbreviation -> TZ ambiguities
addresses https://github.com/karlicoss/HPI/issues/103

for now via experimental time.tz.force_abbreviations config variable
not sure if this whole things is doomed to be resolved properly
2021-03-08 00:40:19 +00:00
Dima Gerasimov
0585cc4a89 arbtt: feed data to influxdb 2021-02-25 19:56:35 +00:00
Dima Gerasimov
ca4d58e4e7 core: add helper to 'freeze' dataclasses, in order to derive a schema from the properties 2021-02-25 19:56:35 +00:00
Dima Gerasimov
86497f9b13 new: basic arbtt module 2021-02-25 19:56:35 +00:00
Dima Gerasimov
20585a3130 influxdb: WIP on magic automatic interface
to run:

    python3 -c 'import my.core.influxdb as I; import my.hypothesis as H; I.magic_fill(H.highlights)'
2021-02-22 10:46:40 +00:00
Dima Gerasimov
bfec6b975f influxdb: add helper to core + use it in bluemaestro/lastfm/rescuetime 2021-02-22 10:46:40 +00:00
Dima Gerasimov
271cd7feef core/cachew: use cache_dir in mcachew if it wasn't specified by the user 2021-02-21 19:51:58 +00:00
Dima Gerasimov
9afe1811a5 core/cachew: special handling for None in order to preserve cache_dir() path
+ add 'suffix' argument for more straighforward logic
2021-02-21 19:51:58 +00:00
Dima Gerasimov
da3c1c9b74 core/cachew: rely on ~/.cache for default cache path
- rely on appdirs for default cache path instead of hardcoded /var/tmp/cachew
  technically backwards incompatible, but no action needed
  you might want to clean /var/tmp/cachew after updating

- use default cache path (e.g. ~/.cache) by default
  see https://github.com/ActiveState/appdirs#some-example-output for more info
  *warning*: things will be cached by default now (used to be uncached before)

- treat cache_dir = None in the config
  *warning*: kind of backwards incompatible.. but again nothing disasterous
2021-02-21 19:51:58 +00:00
Dima Gerasimov
3b4a2a378f core: make discovery even more static, has_stats via ast + tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
f90599d7e4 core: make discovery rely on ast module more, add test 2021-02-19 02:39:25 +00:00
Dima Gerasimov
ddbb2e5f23 CI: better cleanup for modules in between tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
82e2f96192 core: add test for tmp_config; unset new attributes 2021-02-19 02:39:25 +00:00
Dima Gerasimov
5313984d8f core: add tmp_config helper for test & adhoc patching
bluemaestro: cleanup tests
2021-02-19 02:39:25 +00:00
Dima Gerasimov
0534c5c57d cli: add 'hpi module install' and 'hpi module requires'
ci: use hpi module install; remove explicit module links

relevant:

- https://github.com/karlicoss/HPI/issues/12
- https://github.com/karlicoss/HPI/issues/79
2021-02-18 02:04:40 +00:00
Dima Gerasimov
97650adf3b core: add discovery_pure module to get modules and their dependencies via ast module 2021-02-18 02:04:40 +00:00
Dima Gerasimov
4ad4f34cda core: improve mypy coverage 2021-02-18 02:04:40 +00:00
Dima Gerasimov
f102101b39 core/windows: fix get_files and its tests 2021-02-16 06:40:42 +00:00
Dima Gerasimov
746c3da0ca core.pandas: allow specifying schema; add tests 2021-02-15 00:15:44 +00:00
Dima Gerasimov
6239879245 core: add more tests for stat/datetime guessing 2021-02-14 16:20:38 +00:00
Dima Gerasimov
4012f9b7c2 core: more generic functions to jsonify data, rescuetime: fix influxdb filling 2021-02-14 16:20:38 +00:00
Dima Gerasimov
07f901e1e5 core: helpers for automatic dataframes from sequences of NamedTuple/dataclass
also use in my.rescuetime
2021-02-14 16:20:38 +00:00
Dima Gerasimov
df9a7f7390 core.pandas: add check for 'error' column + add empty one by default 2021-02-14 16:20:38 +00:00
Dima Gerasimov
3a1e21635a my.body.blood: use same file
overall I guess this module is highly specific to me anyway, so gonna be
hard to make it generic...
2021-02-14 16:20:38 +00:00
Dima Gerasimov
e81dddddf0 core: proprely resolve class properties in make_config + add test 2020-12-13 18:29:49 +01:00
Dima Gerasimov
571cb48aea core: add modules_ast for more robust module collection 2020-12-11 07:02:16 +01:00
Dima Gerasimov
63c825ab81 my.stackexchange: use GDPR data for votes 2020-12-11 07:02:16 +01:00