Commit graph

820 commits

Author SHA1 Message Date
Sean Breckenridge
39828d07c5 rename RedditThing to RedditBase 2021-10-31 13:15:21 -07:00
Sean Breckenridge
1575422d67 mention my.reddit.pushshift in setup 2021-10-31 13:15:21 -07:00
Sean Breckenridge
6bd625b1a1 remove duplicate save id 2021-10-31 13:15:21 -07:00
Sean Breckenridge
8d1a5736ba add 'raw' properties to protocol members 2021-10-31 13:15:21 -07:00
Sean Breckenridge
299de84919 manually update reddit config instructions 2021-10-31 13:15:21 -07:00
Sean Breckenridge
0be9668024 add docs to pushshift config block 2021-10-31 13:15:21 -07:00
Sean Breckenridge
f6c6bed42d update references to my.reddit across the docs 2021-10-31 13:15:21 -07:00
Sean Breckenridge
6a170b4c10 smarter deprecation warning using regex 2021-10-31 13:15:21 -07:00
Sean Breckenridge
e31d41ca7c warn w/ module name, tell user how to disable warn 2021-10-31 13:15:21 -07:00
Sean Breckenridge
272a5b9b4e simplify reddit.rexport config using migration 2021-10-31 13:15:21 -07:00
Sean Breckenridge
ea14359983 'default's default value is non null 2021-10-31 13:15:21 -07:00
Sean Breckenridge
4492e00250 misc fixes
- convert import_source to a decorator which
  wraps the function call in a try block
- fix protocol class when not TYPE_CHECKING
- add id properties to Protocols, remove attributes
  since protocol expects them to be settable but
  NT is read-only
- use id to merge comments
- remove type: ignore's from reddit config
  and just store as 'Any'
2021-10-31 13:15:21 -07:00
Sean Breckenridge
33f7f48ec5 fix my.config mypy warning 2021-10-31 13:15:21 -07:00
Sean Breckenridge
37cfc412a4 reddit tests imports from my.reddit.rexport 2021-10-31 13:15:21 -07:00
Sean Breckenridge
252037c97b add warning, pushshift install to tests 2021-10-31 13:15:21 -07:00
Sean Breckenridge
a036461daf remove macos 3.6 from CI 2021-10-31 13:15:18 -07:00
Sean Breckenridge
e79f51c25d use lambdas so imports dont run, use __qualname__ 2021-10-31 13:12:03 -07:00
Sean Breckenridge
48865039bf add missing Stats import 2021-10-31 13:12:03 -07:00
Sean Breckenridge
5933711888 initial pushshift/rexport merge implementation 2021-10-31 13:12:03 -07:00
Dima Gerasimov
b54ec0d7f1 ci: fix minor mypy complaints from gitpython 2021-10-29 01:41:44 +01:00
Dima Gerasimov
f5b47dd695 ci: temporary suppress pdfs tests so we can pass CI
see https://github.com/karlicoss/HPI/issues/180
2021-10-29 01:41:44 +01:00
Dima Gerasimov
68d77981db ci: update python stuff, exclude 3.6 from osx 2021-10-29 01:41:44 +01:00
Sean Breckenridge
4a04c09f31 docs: fix copy-paste errors/spelling mistakes 2021-07-10 10:56:23 +01:00
Sean Breckenridge
46198a6447
my.core.serialize: simplejson support, more types (#176)
* my.core.serialize: simplejson support, more types

I added a couple extra checks to the default function,
serializing datetime, dates and dataclasses (incase
orjson isn't installed)

(copied from below)

if orjson couldn't be imported, try simplejson
This is included for compatibility reasons because orjson
is rust-based and compiling on rarer architectures may not work
out of the box

as an example, I've been having issues getting it to install
on my phone (termux/android)

unlike the builtin JSON modue which serializes NamedTuples as lists
(even if you provide a default function), simplejson correctly
serializes namedtuples to dictionaries

this just gives another option to people, simplejson is pure python
so no one should have issues with that. orjson is still way faster,
so still preferable if its easy and theres a precompiled build
for your architecture (which there typically is)

If you're ever running this with simplejson installed and not orjson,
its pretty easy to tell as the JSON styling is different; orjson has
no spaces between tokens, simplejson puts spaces between tokens. e.g.

simplejson: {"a": 5, "b": 10}
orjson: {"a":5,"b":10}
2021-07-08 23:02:56 +01:00
Sean Breckenridge
821bc08a23
core/structure: help locate/extract gdpr exports (#175)
* core/structure: help locate/extract gdpr exports

* ci: add install-types to install stub packages
2021-07-08 00:44:55 +01:00
Dima Gerasimov
8ca88bde2e polar: backward compatibility for my.reading.polar 2021-05-29 13:26:01 +01:00
Dima Gerasimov
2a4bddea79 polar: move to top level, add page support 2021-05-29 13:26:01 +01:00
Sean Breckenridge
e8be20dcb5
core: add tmp_dir for global access to a tmp dir (#173)
* core: add tmp_dir for global access to a tmp dir
2021-05-17 00:28:26 +01:00
Sean Breckenridge
b64a11cc69
smscalls: allow multiple backup dirs (#172)
* smscalls: allow multiple backup dirs
* add smscalls to my.config, add test to CI
2021-05-14 01:35:36 +01:00
Sean Breckenridge
014494059d smscalls: add REQUIRES block to install lxml 2021-05-10 19:51:20 +01:00
Sean Breckenridge
43cfb2742f
cli/query: bugfix, convert output to list (#170)
* cli/query: bugfix, convert output to list to keep it backwards compatible
2021-04-28 21:19:49 +01:00
Sean Breckenridge
fa7474c087 cli/query: add --stream flag
allows you to do something like

hpi query --stream my.reddit.comments
to stream the JSON objects one per line, makes
it nicer to pipe into 'jq'/'fzf' instead
of having to process the giant list
at the end
2021-04-28 18:23:16 +01:00
Sean Breckenridge
d71383ddee stats/is_data_provider: ignore 'inputs' func 2021-04-28 18:00:49 +01:00
Dima Gerasimov
68019c80db core/influx: reuse _locate_functions_or_prompt to choose the data provider 2021-04-27 20:10:10 +01:00
Dima Gerasimov
0517f7ffb8 core/influxdb: add main method to create influx measurement and fill with values
allows running something like

    python3 -m my.core.influxdb populate my.zotero
2021-04-27 20:10:10 +01:00
Sean Breckenridge
0278f2b68d cli/query: improve fallback behaviour/error msg 2021-04-24 06:15:59 +01:00
Dima Gerasimov
491bef83bc bluemaestro: make defensive, yield Exception for measurements 2021-04-22 11:11:39 +01:00
Dima Gerasimov
2611e237a3 my.orgmode: add stat function 2021-04-22 11:11:39 +01:00
Dima Gerasimov
393ed0d9ce core: set _max_workers for dummy concurrent pool 2021-04-22 11:11:39 +01:00
Sean Breckenridge
4b4cb7cb5b cli/query: bugfix where datetime was ignored 2021-04-19 20:21:17 +01:00
Sean Breckenridge
277f0e3988
cli/query: interactive fallback, improve guess_stats (#163) 2021-04-19 18:57:42 +01:00
Dima Gerasimov
91eed15a75 my.zotero: extract top level item's tags 2021-04-13 18:05:49 +01:00
Dima Gerasimov
68d3385468 my.zotero: handle colors & extract human readable 2021-04-13 18:05:49 +01:00
Dima Gerasimov
1ef2c5619e my.zotero: initial version 2021-04-13 18:05:49 +01:00
Sean Breckenridge
c1b70cd90e
docs: add some documentation on module design (#160) 2021-04-11 16:53:43 +01:00
Sean Breckenridge
f559e7cb89 my.coding.commits: fix misspelling/add warning 2021-04-07 19:59:27 +01:00
Sean Breckenridge
fb49243005
core: add hpi query command (#157)
- restructure query code for cli, some test fixes
- initial query_range implementation

    refactored functions in query some more
    to allow re-use in range_range, select()
    pretty much just calls out to a bunch
    of handlers now
2021-04-06 17:19:58 +01:00
Dima Gerasimov
b94120deaf core/sqlite: add compat version for backup() for python3.6 2021-04-05 08:37:07 +01:00
Dima Gerasimov
f09ca17560 core/sqlite: move tests to separate module, pickling during Pool.submit can't handle importing :( 2021-04-05 08:37:07 +01:00
Dima Gerasimov
e99e8725b1 core/sqlite: add a helper to do an im-memory snapshot of db including WAL
+ add a bunch of tests for different WAL behaviours
2021-04-05 08:37:07 +01:00