Commit graph

769 commits

Author SHA1 Message Date
Sean Breckenridge
349ab78fca
core/cli: switch to using click library #155
everything is backwards-compatible with the previous
interface, the only minor changes were to the doctor cmd
which can now accept more than one item to run,
and the --skip-config-check to skip the config_ok
check if the user specifies to

added a test using click.testing.CliRunner (tests
the CLI in an isolated environment), though
additional tests which aren't testing the CLI
itself (parsing arguments or decorator behaviour)
can just call the functions themselves, as they
no longer accept a argparser.Namespace and instead
accept the direct arguments
2021-04-04 10:06:59 +01:00
Dima Gerasimov
5ef2775265 my.github: some work in progress on generating consistent ids
sadly it seems that there are at several issues:

- gdpr has less detailed data so it's hard to generate a proper ID at times
- sometimes there is a small (1s?) discrepancy between created_at between same event in GDPR an API
- some API events can have duplicate payload, but different id, which violates uniqueness
2021-04-02 20:09:53 +01:00
Dima Gerasimov
386234970b my.github.ghexport: handle more event types, more consisten body handling 2021-04-02 20:09:53 +01:00
Dima Gerasimov
b306ccc839 core: add ensure_unique iterator transfromation 2021-04-02 20:09:53 +01:00
Sean Breckenridge
c31641b4eb
core: discovery_pure; allow multiple package roots (#152)
* core: discovery_pure; allow multiple package roots

iterates over my.__path__._path if possible
to discover additional paths to iterate over

else defaults to the path relative to
the current file
2021-04-02 15:46:18 +01:00
Sean Breckenridge
5ecd4b4810 cleanup; remove unused imports 2021-04-02 08:38:06 +01:00
Sean Breckenridge
a11a3af597 commits: reduce possibility of path conflicts 2021-04-02 07:20:35 +01:00
Dima Gerasimov
edf6e5d50b my.pdfs: rely on pdfannots for created date extraction/parsing 2021-04-01 17:27:06 +01:00
Dima Gerasimov
ad177a1ccd my.pdfs: cleanup/refactor
- modernize:
  - add REQUIRES spec for pdfannots library
  - config dataclass/config stub
  - stats function
  - absolute my.core imports in anticipation of splitting core
- use 'paths' instead of 'roots' (better reflects the semantics), use get_files
  backward compatible via config migration
- properly run tests/mypy
2021-04-01 17:27:06 +01:00
Dima Gerasimov
e7604c188e my.pdfs: reorganize tests a bit, fix mypy 2021-04-01 17:27:06 +01:00
Dima Gerasimov
5c38872efc core: add DummyExecutor to make it easier to debug concurrent code with Pools 2021-04-01 17:27:06 +01:00
Sean Breckenridge
3118891c03
my.core.query: initial implementation (#143)
in particular `my.core.query.select`: a function to query, order, sort and filter items from one or more sources
2021-03-28 07:52:50 +01:00
Sean Breckenridge
d47f3c28aa my.core.serialize: support serializing Paths 2021-03-28 06:55:03 +01:00
Sean Breckenridge
1b36bd4379 fix spelling mistakes 2021-03-28 06:53:24 +01:00
Dima Gerasimov
29384aef44 my.goodreads: cleanup, rename from my.reading.goodrads & use proper pip dependency
related:
- https://github.com/karlicoss/HPI/issues/79
- 10d8cc86a1
2021-03-26 05:06:53 +00:00
Sean Breckenridge
1cdef6f40a fix mypy errors
this fixes two distinct mypy errors

one where NamedTuple/dataclassees can't be
defined locally
https://github.com/python/mypy/issues/7281

which happens when you run mypy like
mypy -p my.core on warm cache

the second error is the core/types.py file shadowing the
stdlib types module
2021-03-22 06:34:07 +00:00
ddrone
2bbbff0021 Update README.org 2021-03-21 15:16:21 +00:00
ddrone
8e868391fd Update README.org
*muffled Yung Lean playing in the distance*
2021-03-21 15:16:21 +00:00
Sean Breckenridge
eb26cf8633
my.core.serialize: orjson with additional default and _serialize hook (#140)
basic orjson serialize, json.dumps fallback

Lots of surrounding changes from this discussion:
0593c69056
2021-03-20 00:48:03 +00:00
Sean Breckenridge
02a9fb5e8f github.gdpr: parse project files
also fixed a typo in commit_comments
2021-03-15 12:40:22 +00:00
Dima Gerasimov
a1a24ffbc3 my.coding.commits: more cleanup
Followup of https://github.com/karlicoss/HPI/pull/132

- add REQUIRES section
- use 'commits' config section & add proper schema
- use dedicated subdirectory for cache
2021-03-15 10:33:46 +00:00
Dima Gerasimov
ec8b0e9170 my.coding.commits: actually test on CI, add config stub 2021-03-15 10:33:46 +00:00
Dima Gerasimov
8d6f691824 core: feature: guess module stats from typing annotations 2021-03-15 10:27:18 +00:00
Sean Breckenridge
4db81ca362
cleanup coding.commits (#132)
* cleanup coding.commits

remove the _things check, its never activated
for me and seems pointless

update mechanism for finding fdfind/fd path,
send a core.warning if it fails

update mechanism to cache repos to new cachew
api (remove hashf), cache repos on a per-repo
basis
2021-03-15 03:55:28 +00:00
jon r
44b893a025 add similar projects to README 2021-03-15 03:50:26 +00:00
Sean Breckenridge
fb9426d316 smscalls: add config block
so that don't have to infer what
to set in your hpi config based
on usage in module
2021-03-15 03:48:42 +00:00
Dima Gerasimov
c83bfbd21c ci: enable pull_request trigger 2021-03-10 07:07:38 +00:00
Dima Gerasimov
ce157c47cc google.takeout: support another reported time format
https://github.com/karlicoss/HPI/issues/114
2021-03-08 00:40:19 +00:00
Dima Gerasimov
1fd2a9f643 core/time: more flexible support for resolving TZ abbreviation -> TZ ambiguities
addresses https://github.com/karlicoss/HPI/issues/103

for now via experimental time.tz.force_abbreviations config variable
not sure if this whole things is doomed to be resolved properly
2021-03-08 00:40:19 +00:00
Dima Gerasimov
5ef638694e minor requirements updates 2021-03-08 00:40:19 +00:00
Dima Gerasimov
0585cc4a89 arbtt: feed data to influxdb 2021-02-25 19:56:35 +00:00
Dima Gerasimov
ca4d58e4e7 core: add helper to 'freeze' dataclasses, in order to derive a schema from the properties 2021-02-25 19:56:35 +00:00
Dima Gerasimov
86497f9b13 new: basic arbtt module 2021-02-25 19:56:35 +00:00
Dima Gerasimov
ad924ebca8 refresh readme, reflect blog post changes 2021-02-22 13:29:05 +00:00
Dima Gerasimov
20585a3130 influxdb: WIP on magic automatic interface
to run:

    python3 -c 'import my.core.influxdb as I; import my.hypothesis as H; I.magic_fill(H.highlights)'
2021-02-22 10:46:40 +00:00
Dima Gerasimov
bfec6b975f influxdb: add helper to core + use it in bluemaestro/lastfm/rescuetime 2021-02-22 10:46:40 +00:00
Dima Gerasimov
271cd7feef core/cachew: use cache_dir in mcachew if it wasn't specified by the user 2021-02-21 19:51:58 +00:00
Dima Gerasimov
3e821ca7fd my.github.ghexport: get rid of custom cache_dir 2021-02-21 19:51:58 +00:00
Dima Gerasimov
9afe1811a5 core/cachew: special handling for None in order to preserve cache_dir() path
+ add 'suffix' argument for more straighforward logic
2021-02-21 19:51:58 +00:00
Dima Gerasimov
da3c1c9b74 core/cachew: rely on ~/.cache for default cache path
- rely on appdirs for default cache path instead of hardcoded /var/tmp/cachew
  technically backwards incompatible, but no action needed
  you might want to clean /var/tmp/cachew after updating

- use default cache path (e.g. ~/.cache) by default
  see https://github.com/ActiveState/appdirs#some-example-output for more info
  *warning*: things will be cached by default now (used to be uncached before)

- treat cache_dir = None in the config
  *warning*: kind of backwards incompatible.. but again nothing disasterous
2021-02-21 19:51:58 +00:00
Dima Gerasimov
837ea16dc8 add changelog 2021-02-20 02:06:53 +00:00
Dima Gerasimov
ad1cc71b0f readme: update 2021-02-20 01:39:55 +00:00
Dima Gerasimov
3b4a2a378f core: make discovery even more static, has_stats via ast + tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
f90599d7e4 core: make discovery rely on ast module more, add test 2021-02-19 02:39:25 +00:00
Dima Gerasimov
a3305677b2 core: deprecate my.cfg, instead my.config can (and should be) used directly 2021-02-19 02:39:25 +00:00
Dima Gerasimov
ddbb2e5f23 CI: better cleanup for modules in between tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
94ace823e0 tests: cleanup location/tz tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
82e2f96192 core: add test for tmp_config; unset new attributes 2021-02-19 02:39:25 +00:00
Dima Gerasimov
5313984d8f core: add tmp_config helper for test & adhoc patching
bluemaestro: cleanup tests
2021-02-19 02:39:25 +00:00
Dima Gerasimov
42399f6250 pinboard: *breaking backwards compability*, use pinbexport module directy
Use 'hpi module install my.pinboard' to install it

relevant: https://github.com/karlicoss/HPI/issues/79
2021-02-18 20:46:03 +00:00