Commit graph

599 commits

Author SHA1 Message Date
Dima Gerasimov
92307d5f3d bluemaestro: support new databases as well 2020-07-28 20:32:35 +01:00
Dima Gerasimov
9d45eb0559 bluemaestro: make iterative, add stat() 2020-07-28 20:32:35 +01:00
Tom Craig
fdaae59b59 Add .get to call for d[date] 2020-07-27 21:33:44 +01:00
Dima Gerasimov
092aef88ce core: detect compression, wrap in CPath if necessary 2020-07-26 21:31:26 +01:00
Dima Gerasimov
77deef98de reddit: more consistent handling for events 2020-07-26 21:31:26 +01:00
Dima Gerasimov
031b1278eb reddit: cleanup cachew wrapper a bit 2020-07-26 21:31:26 +01:00
Dima Gerasimov
6b548c24c1 doctor: better mypy detection 2020-07-26 21:31:26 +01:00
Dima Gerasimov
5eecd8721d cli: check specific module with doctor; print help on no command 2020-07-06 21:40:41 +01:00
Dima Gerasimov
49d25a75ae core: use immutable mode in dataset helper 2020-07-06 21:40:41 +01:00
Dima Gerasimov
4fc33a9ed2 core: add helper for opening read-only database 2020-07-06 21:40:41 +01:00
karlicoss
0bcc5952c7
Merge pull request #62 from karlicoss/updates
updates: core & kobo
2020-06-04 22:55:15 +01:00
Dima Gerasimov
821eb47c93 kobo: BREAKING changes. Use kobuddy module directly, rename export_dir to export_path.
Hopefully this makes a lot of sense in the first place, and not that many users, so deserves breaking.
2020-06-04 22:50:52 +01:00
Dima Gerasimov
db852b3927 kobo: move away from my.books 2020-06-04 22:20:48 +01:00
Dima Gerasimov
1cc4eb5d8d core: add helper for computing stats; use it in modules 2020-06-04 22:19:34 +01:00
karlicoss
a94b64c273
Merge pull request #61 from karlicoss/updates
github module: cleanup and proper modular layout
2020-06-01 23:52:07 +01:00
Dima Gerasimov
3d7844b711 core: support '' for explicitly set empty path set 2020-06-01 23:45:26 +01:00
Dima Gerasimov
a267aeec5b github: add config templates + docs
- ghexport: use export_path (export_dir is still supported)
2020-06-01 23:33:34 +01:00
Dima Gerasimov
ca39187c63 github: DEPRECATE my.coding.github
Instead my.github.all should be used (still backward compatible)

The reasons are
a) I don't feel that grouping (i.e. my.coding.*) makes much sense
b) using .all pattern (same way as twitter) allows for more composable and cleaner separation of GDPR and API data
2020-06-01 22:49:31 +01:00
Dima Gerasimov
d7aff1be3f github: start moving to a proper artbitrated module 2020-06-01 22:49:31 +01:00
Matthew Reishus
67cf4d0c04 my.coding.github ignores some events emitted by bots.
I use a service called dependabot ( https://dependabot.com/ ).  It
automatically creates pull requests in my repositories to upgrade
dependencies.  The modern front end javascript world moves really
quickly; projects have a ton of dependencies that are updating all the
time, so there are a lot of these pull requests.

Also, the PRs it makes have a lot of info in them.  Here's an example
one: https://github.com/mreishus/spades/pull/180 .  If you hit the
arrows, you can see it includes a lot of text in "Changelog" and
"Commits".  Now check out the list of closed PRs this project has:
https://github.com/mreishus/spades/pulls?q=is%3Apr+is%3Aclosed

Once I got everything working with my.coding.github, my Github.org
(using orger) was huge: 5MB.  I wanted to get rid of the dependabot
stuff, since it's mostly junk I'm not too interested it, and I got it
down to 130K (from 5MB) just from this commit.

Here's an example of an event I'm filtering out:
I'm looking to see if the "user" contains a "[bot]" tag in it.

  {
    "type": "pull_request",
    "url": "https://github.com/mreishus/spades/pull/96",
    "user": "https://github.com/dependabot-preview[bot]",
    "repository": "https://github.com/mreishus/spades",
    "title": "Bump axios from 0.19.1 to 0.19.2 in /frontend",
    "body": "Bumps [axios](https://github.com/axios/axios) from 0.19.1 to 0.19.2.\n<details>\n<summary>Release notes</summary [cut 5000 characters]
    "base": {
      "ref": "master",
      "sha": "a47687762887c6e5c0d5d0a38c3c9697f09cbcd6",
      "user": "https://github.com/mreishus",
      "repo": "https://github.com/mreishus/spades"
    },
    "head": {
      "ref": "dependabot/npm_and_yarn/frontend/axios-0.19.2",
      "sha": "0e79d0220002cb54cd40e13a40addcc0d0a01482",
      "user": "https://github.com/mreishus",
      "repo": "https://github.com/mreishus/spades"
    },
    "assignee": "https://github.com/mreishus",
    "assignees": [
      "https://github.com/mreishus"
    ],
    "milestone": null,
    "labels": [
      "https://github.com/mreishus/spades/labels/dependencies",
      "https://github.com/mreishus/spades/labels/javascript"
    ],
    "review_requests": [

    ],
    "work_in_progress": false,
    "merged_at": null,
    "closed_at": "2020-01-25T14:40:27Z",
    "created_at": "2020-01-22T13:37:17Z"
  },

Maybe this should be a config option, but I didn't know how to make them
cleanly in HPI, and I'm not sure if anyone would ever want this stuff.
2020-06-01 16:22:07 +01:00
Dima Gerasimov
f175acc848 pocket: reuse pockexport data access layer
BREAKING CHANGE! Data parsing was switched to pockexport.
This would help to keep it consistent across different apps in the future.

When you update, you'll need to:

- clone pockexport (latest version)
- set pockexport repository in your config (see doc/MODULES.org)
2020-05-27 08:42:47 +01:00
Dima Gerasimov
6453ff415d docs: somewhat acceptable data flow diagrams 2020-05-26 22:51:50 +01:00
Dima Gerasimov
150a6a8cb7 docs: wip on better explanation of configs/diagram 2020-05-26 22:51:50 +01:00
karlicoss
04eca6face
Merge pull request #55 from karlicoss/updates
cli updates: doctor mode
2020-05-25 12:30:18 +01:00
Dima Gerasimov
e351c8ba49 cli: add 'config init' command 2020-05-25 12:25:41 +01:00
Dima Gerasimov
7bd7cc9228 cli: integrate with stats reported by the modules 2020-05-25 11:46:30 +01:00
Dima Gerasimov
d890599c7c cli: add checks for importing modules 2020-05-25 11:41:44 +01:00
Dima Gerasimov
8019389ccb cli: move doctor to core, add doc 2020-05-25 10:17:40 +01:00
Dima Gerasimov
dab29a44b5 cli: detect config properly in mypy check 2020-05-25 10:04:58 +01:00
Dima Gerasimov
2ede5b3a5c cli: add config check command 2020-05-25 09:49:57 +01:00
karlicoss
ce8cd5b52c
Merge pull request #54 from karlicoss/updates
core: update warnings, add warn_if_empty decorator fore move defensive data sources
2020-05-25 01:28:42 +01:00
Dima Gerasimov
248e48dc30 core: improve types for warn_if_empty
ok, works with this advice https://github.com/python/mypy/issues/1927 + overloads
2020-05-25 01:23:30 +01:00
Dima Gerasimov
216944b3cd core: improvements for warnings, twitter/rss: try using @warn_if_empty 2020-05-25 00:56:03 +01:00
Dima Gerasimov
616ffb457e core: user overloads to type @warn_if_empty properly.. 2020-05-25 00:25:33 +01:00
Dima Gerasimov
e3a71ea6c6 my.core: more work on typing @warn_if_empty, extra test 2020-05-25 00:25:33 +01:00
Dima Gerasimov
4b22d17188 core: add @warn_if_empty decorator 2020-05-25 00:25:33 +01:00
karlicoss
af814df8e9
Merge pull request #53 from karlicoss/upd
make my.twitter.all easier to override
2020-05-24 23:02:57 +01:00
Dima Gerasimov
f5267d05d7 my.twitter.archive: rename config (preserving bckwd compatibility for now) 2020-05-24 13:06:52 +01:00
Dima Gerasimov
b99b2f3cfa core: add warning when get_files returns no files, my.twitter.archive: make more defensive in case of no archives 2020-05-24 12:51:23 +01:00
Dima Gerasimov
b7662378a2 docs: minor updates 2020-05-22 19:38:14 +01:00
Dima Gerasimov
03773a7b2c twitter module: prettify top level twitter.all 2020-05-22 19:00:02 +01:00
karlicoss
c410daa484
Merge pull request #52 from karlicoss/updates
Updates
2020-05-18 23:40:58 +01:00
Dima Gerasimov
02ba71a91d documentation: generate tables of content, better navigation 2020-05-18 23:31:55 +01:00
Dima Gerasimov
c8bdbfd69f core: expand '~' in get_files & import_dir 2020-05-18 22:43:27 +01:00
Dima Gerasimov
403ec18385 core/modules: get rid of set_repo uses, it was just complicating everythin 2020-05-18 21:33:52 +01:00
Dima Gerasimov
0f80e9d5e6 ok, seems that import_dir is a bit saner 2020-05-18 21:04:38 +01:00
Dima Gerasimov
44aa062756 tests: thinking about external repositories 2020-05-18 20:42:10 +01:00
karlicoss
41c5b34006
Merge pull request #51 from karlicoss/updates
Improve documentation for some modules
2020-05-17 22:10:58 +01:00
Dima Gerasimov
c0bbb4eaf2 misc: get rid of SimpleNamespace uses 2020-05-17 22:05:23 +01:00
Dima Gerasimov
2a9fd54c12 Improve documentation for some modules 2020-05-17 21:56:58 +01:00