HPI/doc/MODULES.org at 0bcc5952c782ef39bb0a43f4667f00c8271ca7d1

fz0x1/HPI

Dima Gerasimov 821eb47c93 kobo: BREAKING changes. Use kobuddy module directly, rename export_dir to export_path.

Hopefully this makes a lot of sense in the first place, and not that many users, so deserves breaking.

2020-06-04 22:50:52 +01:00

9.7 KiB

Raw Blame History

TOC
Intro
Configs

This file is an overview of documented modules (which I'm progressively expanding).

There are many more, see:

"What's inside" for the full list of modules.
you can also run hpi modules to list what's available on your system
source code is always the primary source of truth

If you have some issues with the setup, see "Troubleshooting".

Intro

See SETUP to find out how to set up your own config.

Some explanations:

MY_CONFIG is the path where you are keeping your private configuration (usually ~/.config/my/)
Path is a standard Python object to represent paths
PathIsh is a helper type to allow using either str, or a Path
Paths is another helper type for paths.

It's 'smart', allows you to be flexible about your config:
- simple str or a Path
- /a/path/to/directory/, so the module will consume all files from this directory
- a list of files/directories (it will be flattened)
- a glob string, so you can be flexible about the format of your data on disk (e.g. if you want to keep it compressed)
- empty string (e.g. export_path = ''), this will prevent the module from consuming any data This can be useful for modules that merge multiple data sources (for example, my.twitter or my.github)
Typically, such variable will be passed to get_files to actually extract the list of real files to use. You can see usage examples here.
if the field has a default value, you can omit it from your private config altogether

Configs

The config snippets below are meant to be modified accordingly and pasted into your private configuration, e.g $MY_CONFIG/my/config.py.

You don't have to set them up all at once, it's recommended to do it gradually.

# TODO ugh, pkgutil.walk_packages doesn't recurse and find packages like my.twitter.archive??
# yep.. https://stackoverflow.com/q/41203765/706389
import importlib
# from lint import all_modules # meh
# TODO figure out how to discover configs automatically...
modules = [
    ('google'         , 'my.google.takeout.paths'),
    ('hypothesis'     , 'my.hypothesis'          ),
    ('reddit'         , 'my.reddit'              ),
    ('pocket'         , 'my.pocket'              ),
    ('twint'          , 'my.twitter.twint'       ),
    ('twitter_archive', 'my.twitter.archive'     ),
    ('lastfm'         , 'my.lastfm'              ),
    ('polar'          , 'my.reading.polar'       ),
    ('instapaper'     , 'my.instapaper'          ),
    ('github'         , 'my.github.gdpr'         ),
    ('github'         , 'my.github.ghexport'     ),
    ('kobo'           , 'my.kobo'                ),
]

def indent(s, spaces=4):
    return ''.join(' ' * spaces + l for l in s.splitlines(keepends=True))

from pathlib import Path
import inspect
from dataclasses import fields
import re
print('\n') # ugh. hack for org-ruby drawers bug
for cls, p in modules:
    m = importlib.import_module(p)
    C = getattr(m, cls)
    src = inspect.getsource(C)
    i = src.find('@property')
    if i != -1:
        src = src[:i]
    src = src.strip()
    src = re.sub(r'(class \w+)\(.*', r'\1:', src)
    mpath = p.replace('.', '/')
    for x in ['.py', '__init__.py']:
        if Path(mpath + x).exists():
            mpath = mpath + x
    print(f'** [[file:../{mpath}][{p}]]')
    mdoc = m.__doc__
    if mdoc is not None:
        print(indent(mdoc))
    print(f'    #+begin_src python')
    print(indent(src))
    print(f'    #+end_src')

my.google.takeout.paths

Module for locating and accessing Google Takeout data

class google:
    takeout_path: Paths # path/paths/glob for the takeout zips

my.hypothesis

Hypothes.is highlights and annotations

class hypothesis:
    '''
    Uses [[https://github.com/karlicoss/hypexport][hypexport]] outputs
    '''

    # paths[s]/glob to the exported JSON data
    export_path: Paths

    # path to a local clone of hypexport
    # alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/hypexport
    hypexport  : Optional[PathIsh] = None

my.reddit

Reddit data: saved items/comments/upvotes/etc.

class reddit:
    '''
    Uses [[https://github.com/karlicoss/rexport][rexport]] output.
    '''

    # path[s]/glob to the exported JSON data
    export_path: Paths

    # path to a local clone of rexport
    # alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/rexport
    rexport    : Optional[PathIsh] = None

my.pocket

Pocket bookmarks and highlights

class pocket:
    '''
    Uses [[https://github.com/karlicoss/pockexport][pockexport]] outputs
    '''

    # paths[s]/glob to the exported JSON data
    export_path: Paths

    # path to a local clone of pockexport
    # alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/pockexport
    pockexport  : Optional[PathIsh] = None

my.twitter.twint

Twitter data (tweets and favorites).

Uses Twint data export.

Requirements: pip3 install --user dataset

class twint:
    export_path: Paths # path[s]/glob to the twint Sqlite database

my.twitter.archive

Twitter data (uses official twitter archive export)

class twitter_archive:
    export_path: Paths # path[s]/glob to the twitter archive takeout

my.lastfm

Last.fm scrobbles

class lastfm:
    """
    Uses [[https://github.com/karlicoss/lastfm-backup][lastfm-backup]] outputs
    """
    export_path: Paths

my.reading.polar

Polar articles and highlights

class polar:
    '''
    Polar config is optional, you only need it if you want to specify custom 'polar_dir'
    '''
    polar_dir: PathIsh = Path('~/.polar').expanduser()
    defensive: bool = True # pass False if you want it to fail faster on errors (useful for debugging)

my.instapaper

Instapaper bookmarks, highlights and annotations

class instapaper:
    '''
    Uses [[https://github.com/karlicoss/instapexport][instapexport]] outputs.
    '''
    # path[s]/glob to the exported JSON data
    export_path : Paths

    # path to a local clone of instapexport
    # alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/instapexport
    instapexport: Optional[PathIsh] = None

my.github.gdpr

Github data (uses official GDPR export)

class github:
    gdpr_dir: PathIsh  # path to unpacked GDPR archive

my.github.ghexport

Github data: events, comments, etc. (API data)

class github:
    '''
    Uses [[https://github.com/karlicoss/ghexport][ghexport]] outputs.
    '''
    # path[s]/glob to the exported JSON data
    export_path: Paths

    # path to a local clone of ghexport
    # alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/ghexport
    ghexport : Optional[PathIsh] = None

    # path to a cache directory
    # if omitted, will use /tmp
    cache_dir: Optional[PathIsh] = None

my.kobo

Kobo e-ink reader: annotations and reading stats

class kobo:
    '''
    Uses [[https://github.com/karlicoss/kobuddy#as-a-backup-tool][kobuddy]] outputs.
    '''
    # path[s]/glob to the exported databases
    export_path: Paths

9.7 KiB Raw Blame History

TOC

Intro

Configs

9.7 KiB

Raw Blame History