fz0x1/HPI
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
768 commits 3 branches 41 tags 2.5 MiB
Commit graph

426 commits

Author SHA1 Message Date
Dima Gerasimov
5ef2775265 my.github: some work in progress on generating consistent ids 
sadly it seems that there are at several issues:

- gdpr has less detailed data so it's hard to generate a proper ID at times
- sometimes there is a small (1s?) discrepancy between created_at between same event in GDPR an API
- some API events can have duplicate payload, but different id, which violates uniqueness
 2021-04-02 20:09:53 +01:00
Dima Gerasimov
386234970b my.github.ghexport: handle more event types, more consisten body handling 2021-04-02 20:09:53 +01:00
Dima Gerasimov
b306ccc839 core: add ensure_unique iterator transfromation 2021-04-02 20:09:53 +01:00
Sean Breckenridge
c31641b4eb
core: discovery_pure; allow multiple package roots (#152) 
* core: discovery_pure; allow multiple package roots

iterates over my.__path__._path if possible
to discover additional paths to iterate over

else defaults to the path relative to
the current file
 2021-04-02 15:46:18 +01:00
Sean Breckenridge
5ecd4b4810 cleanup; remove unused imports 2021-04-02 08:38:06 +01:00
Sean Breckenridge
a11a3af597 commits: reduce possibility of path conflicts 2021-04-02 07:20:35 +01:00
Dima Gerasimov
edf6e5d50b my.pdfs: rely on pdfannots for created date extraction/parsing 2021-04-01 17:27:06 +01:00
Dima Gerasimov
ad177a1ccd my.pdfs: cleanup/refactor 
- modernize:
  - add REQUIRES spec for pdfannots library
  - config dataclass/config stub
  - stats function
  - absolute my.core imports in anticipation of splitting core
- use 'paths' instead of 'roots' (better reflects the semantics), use get_files
  backward compatible via config migration
- properly run tests/mypy
 2021-04-01 17:27:06 +01:00
Dima Gerasimov
5c38872efc core: add DummyExecutor to make it easier to debug concurrent code with Pools 2021-04-01 17:27:06 +01:00
Sean Breckenridge
3118891c03
my.core.query: initial implementation (#143) 
in particular `my.core.query.select`: a function to query, order, sort and filter items from one or more sources
 2021-03-28 07:52:50 +01:00
Sean Breckenridge
d47f3c28aa my.core.serialize: support serializing Paths 2021-03-28 06:55:03 +01:00
Sean Breckenridge
1b36bd4379 fix spelling mistakes 2021-03-28 06:53:24 +01:00
Dima Gerasimov
29384aef44 my.goodreads: cleanup, rename from my.reading.goodrads & use proper pip dependency 
related:
- https://github.com/karlicoss/HPI/issues/79
- 10d8cc86a1
 2021-03-26 05:06:53 +00:00
Sean Breckenridge
1cdef6f40a fix mypy errors 
this fixes two distinct mypy errors

one where NamedTuple/dataclassees can't be
defined locally
https://github.com/python/mypy/issues/7281

which happens when you run mypy like
mypy -p my.core on warm cache

the second error is the core/types.py file shadowing the
stdlib types module
 2021-03-22 06:34:07 +00:00
Sean Breckenridge
eb26cf8633
my.core.serialize: orjson with additional default and _serialize hook (#140) 
basic orjson serialize, json.dumps fallback

Lots of surrounding changes from this discussion:
0593c69056
 2021-03-20 00:48:03 +00:00
Sean Breckenridge
02a9fb5e8f github.gdpr: parse project files 
also fixed a typo in commit_comments
 2021-03-15 12:40:22 +00:00
Dima Gerasimov
a1a24ffbc3 my.coding.commits: more cleanup 
Followup of https://github.com/karlicoss/HPI/pull/132

- add REQUIRES section
- use 'commits' config section & add proper schema
- use dedicated subdirectory for cache
 2021-03-15 10:33:46 +00:00
Dima Gerasimov
ec8b0e9170 my.coding.commits: actually test on CI, add config stub 2021-03-15 10:33:46 +00:00
Dima Gerasimov
8d6f691824 core: feature: guess module stats from typing annotations 2021-03-15 10:27:18 +00:00
Sean Breckenridge
4db81ca362
cleanup coding.commits (#132) 
* cleanup coding.commits

remove the _things check, its never activated
for me and seems pointless

update mechanism for finding fdfind/fd path,
send a core.warning if it fails

update mechanism to cache repos to new cachew
api (remove hashf), cache repos on a per-repo
basis
 2021-03-15 03:55:28 +00:00
Sean Breckenridge
fb9426d316 smscalls: add config block 
so that don't have to infer what
to set in your hpi config based
on usage in module
 2021-03-15 03:48:42 +00:00
Dima Gerasimov
ce157c47cc google.takeout: support another reported time format 
https://github.com/karlicoss/HPI/issues/114
 2021-03-08 00:40:19 +00:00
Dima Gerasimov
1fd2a9f643 core/time: more flexible support for resolving TZ abbreviation -> TZ ambiguities 
addresses https://github.com/karlicoss/HPI/issues/103

for now via experimental time.tz.force_abbreviations config variable
not sure if this whole things is doomed to be resolved properly
 2021-03-08 00:40:19 +00:00
Dima Gerasimov
5ef638694e minor requirements updates 2021-03-08 00:40:19 +00:00
Dima Gerasimov
0585cc4a89 arbtt: feed data to influxdb 2021-02-25 19:56:35 +00:00
Dima Gerasimov
ca4d58e4e7 core: add helper to 'freeze' dataclasses, in order to derive a schema from the properties 2021-02-25 19:56:35 +00:00
Dima Gerasimov
86497f9b13 new: basic arbtt module 2021-02-25 19:56:35 +00:00
Dima Gerasimov
20585a3130 influxdb: WIP on magic automatic interface 
to run:

    python3 -c 'import my.core.influxdb as I; import my.hypothesis as H; I.magic_fill(H.highlights)'
 2021-02-22 10:46:40 +00:00
Dima Gerasimov
bfec6b975f influxdb: add helper to core + use it in bluemaestro/lastfm/rescuetime 2021-02-22 10:46:40 +00:00
Dima Gerasimov
271cd7feef core/cachew: use cache_dir in mcachew if it wasn't specified by the user 2021-02-21 19:51:58 +00:00
Dima Gerasimov
3e821ca7fd my.github.ghexport: get rid of custom cache_dir 2021-02-21 19:51:58 +00:00
Dima Gerasimov
9afe1811a5 core/cachew: special handling for None in order to preserve cache_dir() path 
+ add 'suffix' argument for more straighforward logic
 2021-02-21 19:51:58 +00:00
Dima Gerasimov
da3c1c9b74 core/cachew: rely on ~/.cache for default cache path 
- rely on appdirs for default cache path instead of hardcoded /var/tmp/cachew
  technically backwards incompatible, but no action needed
  you might want to clean /var/tmp/cachew after updating

- use default cache path (e.g. ~/.cache) by default
  see https://github.com/ActiveState/appdirs#some-example-output for more info
  *warning*: things will be cached by default now (used to be uncached before)

- treat cache_dir = None in the config
  *warning*: kind of backwards incompatible.. but again nothing disasterous
 2021-02-21 19:51:58 +00:00
Dima Gerasimov
3b4a2a378f core: make discovery even more static, has_stats via ast + tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
f90599d7e4 core: make discovery rely on ast module more, add test 2021-02-19 02:39:25 +00:00
Dima Gerasimov
a3305677b2 core: deprecate my.cfg, instead my.config can (and should be) used directly 2021-02-19 02:39:25 +00:00
Dima Gerasimov
ddbb2e5f23 CI: better cleanup for modules in between tests 2021-02-19 02:39:25 +00:00
Dima Gerasimov
82e2f96192 core: add test for tmp_config; unset new attributes 2021-02-19 02:39:25 +00:00
Dima Gerasimov
5313984d8f core: add tmp_config helper for test & adhoc patching 
bluemaestro: cleanup tests
 2021-02-19 02:39:25 +00:00
Dima Gerasimov
42399f6250 pinboard: *breaking backwards compability*, use pinbexport module directy 
Use 'hpi module install my.pinboard' to install it

relevant: https://github.com/karlicoss/HPI/issues/79
 2021-02-18 20:46:03 +00:00
Dima Gerasimov
0534c5c57d cli: add 'hpi module install' and 'hpi module requires' 
ci: use hpi module install; remove explicit module links

relevant:

- https://github.com/karlicoss/HPI/issues/12
- https://github.com/karlicoss/HPI/issues/79
 2021-02-18 02:04:40 +00:00
Dima Gerasimov
97650adf3b core: add discovery_pure module to get modules and their dependencies via ast module 2021-02-18 02:04:40 +00:00
Dima Gerasimov
4ad4f34cda core: improve mypy coverage 2021-02-18 02:04:40 +00:00
Dima Gerasimov
f102101b39 core/windows: fix get_files and its tests 2021-02-16 06:40:42 +00:00
Dima Gerasimov
6d9bc2964b bluemaestro: populate grafana 2021-02-15 00:15:44 +00:00
Dima Gerasimov
1899b006de bluemaestro: investigation of data quality + more sanity checks 2021-02-15 00:15:44 +00:00
Dima Gerasimov
746c3da0ca core.pandas: allow specifying schema; add tests 2021-02-15 00:15:44 +00:00
Dima Gerasimov
d77ab92d86 bluemaesto: get rid of unnecessary file, move to top level 2021-02-15 00:15:44 +00:00
Dima Gerasimov
6239879245 core: add more tests for stat/datetime guessing 2021-02-14 16:20:38 +00:00
Dima Gerasimov
4012f9b7c2 core: more generic functions to jsonify data, rescuetime: fix influxdb filling 2021-02-14 16:20:38 +00:00