[[https://circleci.com/gh/karlicoss/my/tree/master][https://circleci.com/gh/karlicoss/my/tree/master.svg?style=svg]] Python interface into my life. In short, this package provides programmatic access to my personal data and knowledge. Gory details of getting data, parsing, etc. are abstracted away and you get nice and familiar Python objects. It makes it easier to access, work with, analyze and combine data and leverage on existing libraries for data analysis like Pandas, Matplotlib, etc. This particular setup might not necessarily be most convenient for you to use, perhaps it's more of a concept of how you can organize, access and use personal data. But it definitely works for me, so hopefully that would help you and serve as as source of inspiration. The readme is more of a setup manual, I'm writing about motivation and specific usecases [[https://beepb00p.xyz/mypkg.html][here]]. Short example to give you an idea: "which subreddits I find most interesting?" #+begin_src python :python "with_my python3" :exports both from my.reddit import get_saves from collections import Counter saves = get_saves() return Counter(s.subreddit for s in saves).most_common(3) #+end_src #+RESULTS: | orgmode | 46 | | AskReddit | 31 | | QuantifiedSelf | 30 | * Supported modules #+begin_src python :results output table drawer :exports results :python "with_my python3" from pathlib import Path import re import importlib def ignored(m: str): excluded = [ 'kython.*', 'bluemaestro.check', 'body', 'books', 'calendar', 'coding', 'coding.codeforces', 'coding.topcoder', 'media', 'mycfg_stub', 'reading', 'takeout', '_rss', 'common', 'error', ] exs = '|'.join(excluded) return re.match(f'^my.({exs})$', m) for f in list(sorted(Path('my/').glob('**/*.py'))): if f.is_symlink(): continue # meh if f.name == '__init__.py': f = f.parent m = str(f.with_suffix('')).replace('/', '.') if ignored(m): continue # TODO module link? # TODO I've done this for infra diagram already... mod = importlib.import_module(m) doc = mod.__doc__ if doc is None: pass # TODO # print(m, ": NO DOCS!") continue else: fline = doc.strip().splitlines()[0] mlink = f'[[{f}][{m}]]' print('|', mlink, '|', fline, '|') #+end_src #+RESULTS: :results: | [[my/bluemaestro][my.bluemaestro]] | Bluemaestro temperature/humidity/pressure monitor | | [[my/body/blood.py][my.body.blood]] | Blood tracking | | [[my/books/kobo.py][my.books.kobo]] | Kobo e-ink reader: annotations and reading stats | | [[my/calendar/holidays.py][my.calendar.holidays]] | Provides data on days off work (based on public holidays + manual inputs) | | [[my/coding/github.py][my.coding.github]] | Github events and their metadata: comments/issues/pull requests | | [[my/fbmessenger.py][my.fbmessenger]] | Module for Facebook Messenger messages | | [[my/feedbin.py][my.feedbin]] | Module for Feedbin RSS reader | | [[my/feedly.py][my.feedly]] | Module for Fedly RSS reader | | [[my/hypothesis.py][my.hypothesis]] | Hypothes.is highlights and annotations | | [[my/instapaper.py][my.instapaper]] | Instapaper bookmarks, highlights and annotations | | [[my/location/takeout.py][my.location.takeout]] | Module for Google Takeout data | | [[my/materialistic.py][my.materialistic]] | Module for [[https://play.google.com/store/apps/details?id=io.github.hidroh.materialistic][Materialistic]] app for Hackernews | | [[my/pinboard.py][my.pinboard]] | Module for pinboard.in bookmarks | | [[my/reading/polar.py][my.reading.polar]] | Module for Polar articles and highlights | | [[my/reddit.py][my.reddit]] | Module for Reddit data: saved items/comments/upvotes etc | | [[my/twitter.py][my.twitter]] | Module for Twitter (uses official twitter archive export) | :end: * Setting up ** =mycfg= package for private paths/repositories (optional) If you're not planning to use private configuration (some modules don't need it) you can skip straight to the next step. Still, I'd recommend you to read anyway. First you need to tell the package where to look for your data and external repositories, which is done though a separate (private) package named ~mycfg~. You can see example in ~mycfg_template~. You can copy it somewhere else and modify to your needs. Some explanations: #+begin_src bash :exports results :results output for x in $(find mycfg_template/ | grep -v -E 'mypy_cache|.git|__pycache__|scignore'); do if [[ -L "$x" ]]; then echo "l $x -> $(readlink $x)" elif [[ -d "$x" ]]; then echo "d $x" else echo "f $x" (echo "---"; cat "$x"; echo "---" ) | sed 's/^/ /' fi done #+end_src #+RESULTS: #+begin_example d mycfg_template/ d mycfg_template/mycfg f mycfg_template/mycfg/__init__.py --- class paths: """ Feel free to remove this if you don't need it/add your own custom settings and use them """ class hypothesis: export_path = '/tmp/my_demo/backups/hypothesis' --- d mycfg_template/mycfg/repos l mycfg_template/mycfg/repos/hypexport -> /tmp/my_demo/hypothesis_repo #+end_example As you can see, generally you specify fixed paths (e.g. to backup directory) in ~__init__.py~. Feel free to add other files as well though to organize better, it's a real python package after all! Some things (e.g. links to external packages like [[https://github.com/karlicoss/hypexport][hypexport]]) are specified as normal symlinks in ~repos~ directory. That way you get easy imports (e.g. =import mycfg.repos.hypexport.model=) and proper IDE integration. # TODO link to post about exports? ** =with_my= helper script Next, point =with_my= script to your private configuration: #+begin_src bash cp with_my.example with_my vim with_my # specify path to your mycfg (if you want to use it) #+end_src It's also convenient to put =with_my= somewhere in your system path so you can run it from anywhere. ** Dependencies Dependencies are different for specific modules you're planning to use, so it's hard to specify. Generally you can just try using the module and then install missing packages via ~pip install --user~, should be fairly straightforward. * Usage examples If you run your script with ~with_my~ wrapper, you'd have ~my~ in ~PYTHONPATH~ which gives you access to your data from within the script. - accessing Kobo books #+begin_src bash with_my python3 -c 'import my.books.kobo as kobo; print(kobo.get_todos())' #+end_src - if you have [[https://github.com/karlicoss/orger][orger]] installed, you can use its modules to get Org-mode representations of your data. For instance, rendering [[https://github.com/burtonator/polar-bookshelf][Polar]] highlights as org-mode file as easy as: #+begin_src bash with_my orger/modules/polar.py --to polar.org #+end_src - read/run [[./demo.py][demo.py]] for a full demonstration of setting up Hypothesis (it uses public annotations data from Github) * Linting #+begin_src bash # see https://github.com/python/mypy/issues/1645 for --namespace-packages explanation with_my mypy --namespace-packages my #+end_src or, set up as ~mypy.ini~ file: #+begin_src [mypy] mypy_path=/path/to/mycfg_dir #+end_src