fz0x1/HPI
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
826 commits 3 branches 41 tags 2.5 MiB
Commit graph

479 commits

Author SHA1 Message Date
Dima Gerasimov
0e891a267f doctor: suggest config documentation in case of ImportError from config 
doesn't help in all cases but perhaps helpful anyway

relevant: https://github.com/karlicoss/HPI/issues/109
 2022-02-02 23:46:46 +00:00
Dima Gerasimov
d1f791dee8 my.fbmessenger: move fbmessenger.py into fbmessenger/export.py 
keeping it backwards compatible + conditional warning similar to https://github.com/karlicoss/HPI/pull/179

follow up for https://github.com/seanbreckenridge/HPI/pull/18
for now without the __path__ hacking, will do it in bulk later

too lazy to run test_import_warnings.sh on CI for now, but figured I'd commit it for the reference anyway
 2022-02-02 23:22:45 +00:00
Dima Gerasimov
e30953195c instagram: initial module for android app data (direct messages) 2022-02-02 21:50:43 +00:00
Sean Breckenridge
823668ca5c make reddit.rexport logs info by default 
can always be configured with HPI_LOGS
having this on debug makes hpi doctor
quite verbose
 2022-02-02 00:35:54 +00:00
Dima Gerasimov
7ead8eb4c9 bumble: add initial module for android database 2022-01-30 23:56:24 +00:00
Dima Gerasimov
673ee53a49 my.zulip: add message permalink 2022-01-30 23:33:05 +00:00
Dima Gerasimov
a39b5605ae my.zulip: extract Server/Sender objects, experiment with normalised and denormalised objects 2022-01-30 23:33:05 +00:00
Dima Gerasimov
a1f03f9c02 my.zulip: initial zulip module, parsing full public organization export archive 2022-01-27 22:58:33 +00:00
Dima Gerasimov
73c9e46c4c core: better support for compressed stuff, add .tar.gz 2022-01-27 22:58:33 +00:00
Sean Breckenridge
7493770d4d core: remove vendorized py37 isoformat code 2022-01-27 19:25:42 +00:00
Sean Breckenridge
03dd1271f4 cli/query: add short flags, stream affects pprint 
adds some short flags as CLI flags for convenience
the --stream flag previously only affected json, but
I can imagine '-o pprint -s -l 5' to print the first
5 items from some function could be useful as well
 2022-01-27 08:50:57 +00:00
Sean Breckenridge
3f4fb64d56
core: drop py36 support, update docs for reddit (#193) 
* docs: update references to my.reddit
* ci: remove 3.6, add 3.9
 2022-01-27 08:26:15 +00:00
Dima Gerasimov
be21606075 my.reddit: better handling for legacy reddit config 
prior to this change it would error with

    @dataclass
>   class pushshift_config(uconfig.pushshift):
E   AttributeError: type object 'test_config' has no attribute 'pushshift'
 2021-12-24 18:02:37 +00:00
Dima Gerasimov
5e9cc2a6a0 my.reddit: enable CI tests 2021-12-24 18:02:37 +00:00
Sean Breckenridge
01dfbbd58e use default for getattr instead of catching error 2021-12-19 19:33:31 +00:00
Sean Breckenridge
83725e49dd cli/query: allow querying dynamic functions 2021-12-19 19:33:31 +00:00
Dima Gerasimov
dd928964e6 general: fix mypy errors after mypy and pytz stubs updates 
see 968fd6d01d/stubs/pytz/pytz/tzinfo.pyi (L6)
it says all concrete instances should not be None
 2021-12-19 18:53:29 +00:00
Dima Gerasimov
9578b13fca my.pdf: handle update to pdfannots 0.2 
undoes f5b47dd695 , tests work properly now

resolves https://github.com/karlicoss/HPI/issues/180
 2021-12-19 18:53:29 +00:00
Sean Breckenridge
074b8685d6 reddit: pass logger to cachew 
so that HPI_LOGS can be used to interact
with this module, to check if cachew
is working properly
 2021-12-19 18:25:50 +00:00
Sean Breckenridge
d006339ab4 reddit: fix spelling mistakes 2021-11-03 20:18:10 +00:00
Sean Breckenridge
d6c484f321 reddit: ensure rexport isnt pointing to repo 2021-10-31 21:47:10 +00:00
Sean Breckenridge
5d2eadbcc6
reddit: swap inheritance order for Protocol (#183) 2021-10-31 21:24:16 +00:00
Sean Breckenridge
8422c6e420
my.reddit: refactor into module that supports pushshift/gdpr (#179) 
* initial pushshift/rexport merge implementation, using id for merging
* smarter module deprecation warning using regex
* add `RedditBase` from promnesia
* `import_source` helper for gracefully handing mixin data sources
 2021-10-31 20:39:04 +00:00
Dima Gerasimov
b54ec0d7f1 ci: fix minor mypy complaints from gitpython 2021-10-29 01:41:44 +01:00
Sean Breckenridge
4a04c09f31 docs: fix copy-paste errors/spelling mistakes 2021-07-10 10:56:23 +01:00
Sean Breckenridge
46198a6447
my.core.serialize: simplejson support, more types (#176) 
* my.core.serialize: simplejson support, more types

I added a couple extra checks to the default function,
serializing datetime, dates and dataclasses (incase
orjson isn't installed)

(copied from below)

if orjson couldn't be imported, try simplejson
This is included for compatibility reasons because orjson
is rust-based and compiling on rarer architectures may not work
out of the box

as an example, I've been having issues getting it to install
on my phone (termux/android)

unlike the builtin JSON modue which serializes NamedTuples as lists
(even if you provide a default function), simplejson correctly
serializes namedtuples to dictionaries

this just gives another option to people, simplejson is pure python
so no one should have issues with that. orjson is still way faster,
so still preferable if its easy and theres a precompiled build
for your architecture (which there typically is)

If you're ever running this with simplejson installed and not orjson,
its pretty easy to tell as the JSON styling is different; orjson has
no spaces between tokens, simplejson puts spaces between tokens. e.g.

simplejson: {"a": 5, "b": 10}
orjson: {"a":5,"b":10}
 2021-07-08 23:02:56 +01:00
Sean Breckenridge
821bc08a23
core/structure: help locate/extract gdpr exports (#175) 
* core/structure: help locate/extract gdpr exports

* ci: add install-types to install stub packages
 2021-07-08 00:44:55 +01:00
Dima Gerasimov
8ca88bde2e polar: backward compatibility for my.reading.polar 2021-05-29 13:26:01 +01:00
Dima Gerasimov
2a4bddea79 polar: move to top level, add page support 2021-05-29 13:26:01 +01:00
Sean Breckenridge
e8be20dcb5
core: add tmp_dir for global access to a tmp dir (#173) 
* core: add tmp_dir for global access to a tmp dir
 2021-05-17 00:28:26 +01:00
Sean Breckenridge
b64a11cc69
smscalls: allow multiple backup dirs (#172) 
* smscalls: allow multiple backup dirs
* add smscalls to my.config, add test to CI
 2021-05-14 01:35:36 +01:00
Sean Breckenridge
014494059d smscalls: add REQUIRES block to install lxml 2021-05-10 19:51:20 +01:00
Sean Breckenridge
43cfb2742f
cli/query: bugfix, convert output to list (#170) 
* cli/query: bugfix, convert output to list to keep it backwards compatible
 2021-04-28 21:19:49 +01:00
Sean Breckenridge
fa7474c087 cli/query: add --stream flag 
allows you to do something like

hpi query --stream my.reddit.comments
to stream the JSON objects one per line, makes
it nicer to pipe into 'jq'/'fzf' instead
of having to process the giant list
at the end
 2021-04-28 18:23:16 +01:00
Sean Breckenridge
d71383ddee stats/is_data_provider: ignore 'inputs' func 2021-04-28 18:00:49 +01:00
Dima Gerasimov
68019c80db core/influx: reuse _locate_functions_or_prompt to choose the data provider 2021-04-27 20:10:10 +01:00
Dima Gerasimov
0517f7ffb8 core/influxdb: add main method to create influx measurement and fill with values 
allows running something like

    python3 -m my.core.influxdb populate my.zotero
 2021-04-27 20:10:10 +01:00
Sean Breckenridge
0278f2b68d cli/query: improve fallback behaviour/error msg 2021-04-24 06:15:59 +01:00
Dima Gerasimov
491bef83bc bluemaestro: make defensive, yield Exception for measurements 2021-04-22 11:11:39 +01:00
Dima Gerasimov
2611e237a3 my.orgmode: add stat function 2021-04-22 11:11:39 +01:00
Dima Gerasimov
393ed0d9ce core: set _max_workers for dummy concurrent pool 2021-04-22 11:11:39 +01:00
Sean Breckenridge
4b4cb7cb5b cli/query: bugfix where datetime was ignored 2021-04-19 20:21:17 +01:00
Sean Breckenridge
277f0e3988
cli/query: interactive fallback, improve guess_stats (#163) 2021-04-19 18:57:42 +01:00
Dima Gerasimov
91eed15a75 my.zotero: extract top level item's tags 2021-04-13 18:05:49 +01:00
Dima Gerasimov
68d3385468 my.zotero: handle colors & extract human readable 2021-04-13 18:05:49 +01:00
Dima Gerasimov
1ef2c5619e my.zotero: initial version 2021-04-13 18:05:49 +01:00
Sean Breckenridge
f559e7cb89 my.coding.commits: fix misspelling/add warning 2021-04-07 19:59:27 +01:00
Sean Breckenridge
fb49243005
core: add hpi query command (#157) 
- restructure query code for cli, some test fixes
- initial query_range implementation

    refactored functions in query some more
    to allow re-use in range_range, select()
    pretty much just calls out to a bunch
    of handlers now
 2021-04-06 17:19:58 +01:00
Dima Gerasimov
b94120deaf core/sqlite: add compat version for backup() for python3.6 2021-04-05 08:37:07 +01:00
Dima Gerasimov
f09ca17560 core/sqlite: move tests to separate module, pickling during Pool.submit can't handle importing :( 2021-04-05 08:37:07 +01:00