9.2 KiB
This file is an overview of documented modules (which I'm progressively expanding).
There are many more, see:
- "What's inside" for the full list of modules.
- you can also run
hpi modulesto list what's available on your system - source code is always the primary source of truth
If you have some issues with the setup, see "Troubleshooting".
TOC
Intro
See SETUP to find out how to set up your own config.
Some explanations:
MY_CONFIGis the path where you are keeping your private configuration (usually~/.config/my/)- Path is a standard Python object to represent paths
- PathIsh is a helper type to allow using either
str, or aPath -
Paths is another helper type for paths.
It's 'smart', allows you to be flexible about your config:
- simple
stror aPath /a/path/to/directory/, so the module will consume all files from this directory- a list of files/directories (it will be flattened)
- a glob string, so you can be flexible about the format of your data on disk (e.g. if you want to keep it compressed)
- empty sequence (e.g.
export_path = ()), this is useful for modules that merge multiple data sources (for example,my.twitter)
Typically, such variable will be passed to
get_filesto actually extract the list of real files to use. You can see usage examples here. - simple
- if the field has a default value, you can omit it from your private config altogether
Configs
The config snippets below are meant to be modified accordingly and pasted into your private configuration, e.g $MY_CONFIG/my/config.py.
You don't have to set them up all at once, it's recommended to do it gradually.
# TODO ugh, pkgutil.walk_packages doesn't recurse and find packages like my.twitter.archive??
# yep.. https://stackoverflow.com/q/41203765/706389
import importlib
# from lint import all_modules # meh
# TODO figure out how to discover configs automatically...
modules = [
('google' , 'my.google.takeout.paths'),
('hypothesis' , 'my.hypothesis' ),
('reddit' , 'my.reddit' ),
('pocket' , 'my.pocket' ),
('twint' , 'my.twitter.twint' ),
('twitter_archive', 'my.twitter.archive' ),
('lastfm' , 'my.lastfm' ),
('polar' , 'my.reading.polar' ),
('instapaper' , 'my.instapaper' ),
('github' , 'my.github.gdpr' ),
('github' , 'my.github.ghexport' ),
]
def indent(s, spaces=4):
return ''.join(' ' * spaces + l for l in s.splitlines(keepends=True))
from pathlib import Path
import inspect
from dataclasses import fields
import re
print('\n') # ugh. hack for org-ruby drawers bug
for cls, p in modules:
m = importlib.import_module(p)
C = getattr(m, cls)
src = inspect.getsource(C)
i = src.find('@property')
if i != -1:
src = src[:i]
src = src.strip()
src = re.sub(r'(class \w+)\(.*', r'\1:', src)
mpath = p.replace('.', '/')
for x in ['.py', '__init__.py']:
if Path(mpath + x).exists():
mpath = mpath + x
print(f'** [[file:../{mpath}][{p}]]')
mdoc = m.__doc__
if mdoc is not None:
print(indent(mdoc))
print(f' #+begin_src python')
print(indent(src))
print(f' #+end_src')my.google.takeout.paths
Module for locating and accessing Google Takeout data
class google:
takeout_path: Paths # path/paths/glob for the takeout zipsmy.hypothesis
Hypothes.is highlights and annotations
class hypothesis:
'''
Uses [[https://github.com/karlicoss/hypexport][hypexport]] outputs
'''
# paths[s]/glob to the exported JSON data
export_path: Paths
# path to a local clone of hypexport
# alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/hypexport
hypexport : Optional[PathIsh] = Nonemy.reddit
Reddit data: saved items/comments/upvotes/etc.
class reddit:
'''
Uses [[https://github.com/karlicoss/rexport][rexport]] output.
'''
# path[s]/glob to the exported JSON data
export_path: Paths
# path to a local clone of rexport
# alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/rexport
rexport : Optional[PathIsh] = Nonemy.pocket
Pocket bookmarks and highlights
class pocket:
'''
Uses [[https://github.com/karlicoss/pockexport][pockexport]] outputs
'''
# paths[s]/glob to the exported JSON data
export_path: Paths
# path to a local clone of pockexport
# alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/pockexport
pockexport : Optional[PathIsh] = Nonemy.twitter.twint
Twitter data (tweets and favorites).
Uses Twint data export.
Requirements: pip3 install --user dataset
class twint:
export_path: Paths # path[s]/glob to the twint Sqlite databasemy.twitter.archive
Twitter data (uses official twitter archive export)
class twitter_archive:
export_path: Paths # path[s]/glob to the twitter archive takeoutmy.lastfm
Last.fm scrobbles
class lastfm:
"""
Uses [[https://github.com/karlicoss/lastfm-backup][lastfm-backup]] outputs
"""
export_path: Pathsmy.reading.polar
Polar articles and highlights
class polar:
'''
Polar config is optional, you only need it if you want to specify custom 'polar_dir'
'''
polar_dir: PathIsh = Path('~/.polar').expanduser()
defensive: bool = True # pass False if you want it to fail faster on errors (useful for debugging)my.instapaper
Instapaper bookmarks, highlights and annotations
class instapaper:
'''
Uses [[https://github.com/karlicoss/instapexport][instapexport]] outputs.
'''
# path[s]/glob to the exported JSON data
export_path : Paths
# path to a local clone of instapexport
# alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/instapexport
instapexport: Optional[PathIsh] = Nonemy.github.gdpr
Github data (uses official GDPR export)
class github:
gdpr_dir: PathIsh # path to unpacked GDPR archivemy.github.ghexport
Github data: events, comments, etc. (API data)
class github:
'''
Uses [[https://github.com/karlicoss/ghexport][ghexport]] outputs.
'''
# path[s]/glob to the exported JSON data
export_path: Paths
# path to a local clone of ghexport
# alternatively, you can put the repository (or a symlink) in $MY_CONFIG/my/config/repos/ghexport
ghexport : Optional[PathIsh] = None
# path to a cache directory
# if omitted, will use /tmp
cache_dir: Optional[PathIsh] = None