move setup file to doc/
This commit is contained in:
Dima Gerasimov
parent
6c5d713a17
commit
4cd85066e5
1 changed files with 0 additions and 0 deletions
190
doc/SETUP.org
Normal file
190
doc/SETUP.org
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
[[https://circleci.com/gh/karlicoss/my/tree/master][https://circleci.com/gh/karlicoss/my/tree/master.svg?style=svg]]
|
||||
|
||||
Python interface into my life.
|
||||
|
||||
In short, this package provides programmatic access to my personal data and knowledge.
|
||||
Gory details of getting data, parsing, etc. are abstracted away and you get nice and familiar Python objects.
|
||||
It makes it easier to access, work with, analyze and combine data and leverage on existing libraries for data analysis like Pandas, Matplotlib, etc.
|
||||
|
||||
This particular setup might not necessarily be most convenient for you to use, perhaps it's more of a concept of how you can organize, access and use personal data.
|
||||
But it definitely works for me, so hopefully that would help you and serve as as source of inspiration.
|
||||
|
||||
The readme is more of a setup manual, I'm writing about motivation and specific usecases [[https://beepb00p.xyz/mypkg.html][here]].
|
||||
Short example to give you an idea: "which subreddits I find most interesting?"
|
||||
|
||||
#+begin_src python :python "with_my python3" :exports both
|
||||
from my.reddit import get_saves
|
||||
from collections import Counter
|
||||
saves = get_saves()
|
||||
return Counter(s.subreddit for s in saves).most_common(3)
|
||||
#+end_src
|
||||
|
||||
#+RESULTS:
|
||||
| orgmode | 46 |
|
||||
| AskReddit | 31 |
|
||||
| QuantifiedSelf | 30 |
|
||||
|
||||
* Supported modules
|
||||
|
||||
#+begin_src python :results output table drawer :exports results :python "with_my python3"
|
||||
from pathlib import Path
|
||||
import re
|
||||
import importlib
|
||||
|
||||
def ignored(m: str):
|
||||
excluded = [
|
||||
'kython.*',
|
||||
'bluemaestro.check',
|
||||
'body',
|
||||
'books',
|
||||
'calendar',
|
||||
'coding',
|
||||
'coding.codeforces',
|
||||
'coding.topcoder',
|
||||
'media',
|
||||
'mycfg_stub',
|
||||
'reading',
|
||||
'takeout',
|
||||
'_rss',
|
||||
'common',
|
||||
'error',
|
||||
]
|
||||
exs = '|'.join(excluded)
|
||||
return re.match(f'^my.({exs})$', m)
|
||||
|
||||
for f in list(sorted(Path('my/').glob('**/*.py'))):
|
||||
if f.is_symlink():
|
||||
continue # meh
|
||||
if f.name == '__init__.py':
|
||||
f = f.parent
|
||||
m = str(f.with_suffix('')).replace('/', '.')
|
||||
if ignored(m):
|
||||
continue
|
||||
# TODO module link?
|
||||
# TODO I've done this for infra diagram already...
|
||||
mod = importlib.import_module(m)
|
||||
doc = mod.__doc__
|
||||
if doc is None:
|
||||
pass # TODO
|
||||
# print(m, ": NO DOCS!")
|
||||
continue
|
||||
else:
|
||||
fline = doc.strip().splitlines()[0]
|
||||
mlink = f'[[{f}][{m}]]'
|
||||
print('|', mlink, '|', fline, '|')
|
||||
|
||||
#+end_src
|
||||
|
||||
#+RESULTS:
|
||||
:results:
|
||||
| [[my/bluemaestro][my.bluemaestro]] | Bluemaestro temperature/humidity/pressure monitor |
|
||||
| [[my/body/blood.py][my.body.blood]] | Blood tracking |
|
||||
| [[my/books/kobo.py][my.books.kobo]] | Kobo e-ink reader: annotations and reading stats |
|
||||
| [[my/calendar/holidays.py][my.calendar.holidays]] | Provides data on days off work (based on public holidays + manual inputs) |
|
||||
| [[my/coding/github.py][my.coding.github]] | Github events and their metadata: comments/issues/pull requests |
|
||||
| [[my/fbmessenger.py][my.fbmessenger]] | Module for Facebook Messenger messages |
|
||||
| [[my/feedbin.py][my.feedbin]] | Module for Feedbin RSS reader |
|
||||
| [[my/feedly.py][my.feedly]] | Module for Fedly RSS reader |
|
||||
| [[my/hypothesis.py][my.hypothesis]] | Hypothes.is highlights and annotations |
|
||||
| [[my/instapaper.py][my.instapaper]] | Instapaper bookmarks, highlights and annotations |
|
||||
| [[my/location/takeout.py][my.location.takeout]] | Module for Google Takeout data |
|
||||
| [[my/materialistic.py][my.materialistic]] | Module for [[https://play.google.com/store/apps/details?id=io.github.hidroh.materialistic][Materialistic]] app for Hackernews |
|
||||
| [[my/pinboard.py][my.pinboard]] | Module for pinboard.in bookmarks |
|
||||
| [[my/reading/polar.py][my.reading.polar]] | Module for Polar articles and highlights |
|
||||
| [[my/reddit.py][my.reddit]] | Module for Reddit data: saved items/comments/upvotes etc |
|
||||
| [[my/twitter.py][my.twitter]] | Module for Twitter (uses official twitter archive export) |
|
||||
:end:
|
||||
|
||||
|
||||
|
||||
* Setting up
|
||||
** =mycfg= package for private paths/repositories (optional)
|
||||
If you're not planning to use private configuration (some modules don't need it) you can skip straight to the next step. Still, I'd recommend you to read anyway.
|
||||
|
||||
First you need to tell the package where to look for your data and external repositories, which is done though a separate (private) package named ~mycfg~.
|
||||
|
||||
You can see example in ~mycfg_template~. You can copy it somewhere else and modify to your needs.
|
||||
|
||||
Some explanations:
|
||||
|
||||
#+begin_src bash :exports results :results output
|
||||
for x in $(find mycfg_template/ | grep -v -E 'mypy_cache|.git|__pycache__|scignore'); do
|
||||
if [[ -L "$x" ]]; then
|
||||
echo "l $x -> $(readlink $x)"
|
||||
elif [[ -d "$x" ]]; then
|
||||
echo "d $x"
|
||||
else
|
||||
echo "f $x"
|
||||
(echo "---"; cat "$x"; echo "---" ) | sed 's/^/ /'
|
||||
fi
|
||||
done
|
||||
#+end_src
|
||||
|
||||
#+RESULTS:
|
||||
#+begin_example
|
||||
d mycfg_template/
|
||||
d mycfg_template/mycfg
|
||||
f mycfg_template/mycfg/__init__.py
|
||||
---
|
||||
class paths:
|
||||
"""
|
||||
Feel free to remove this if you don't need it/add your own custom settings and use them
|
||||
"""
|
||||
class hypothesis:
|
||||
export_path = '/tmp/my_demo/backups/hypothesis'
|
||||
---
|
||||
d mycfg_template/mycfg/repos
|
||||
l mycfg_template/mycfg/repos/hypexport -> /tmp/my_demo/hypothesis_repo
|
||||
#+end_example
|
||||
|
||||
As you can see, generally you specify fixed paths (e.g. to backup directory) in ~__init__.py~.
|
||||
Feel free to add other files as well though to organize better, it's a real python package after all!
|
||||
|
||||
Some things (e.g. links to external packages like [[https://github.com/karlicoss/hypexport][hypexport]]) are specified as normal symlinks in ~repos~ directory.
|
||||
That way you get easy imports (e.g. =import mycfg.repos.hypexport.model=) and proper IDE integration.
|
||||
|
||||
# TODO link to post about exports?
|
||||
** =with_my= helper script
|
||||
Next, point =with_my= script to your private configuration:
|
||||
|
||||
#+begin_src bash
|
||||
cp with_my.example with_my
|
||||
vim with_my # specify path to your mycfg (if you want to use it)
|
||||
#+end_src
|
||||
|
||||
It's also convenient to put =with_my= somewhere in your system path so you can run it from anywhere.
|
||||
|
||||
** Dependencies
|
||||
Dependencies are different for specific modules you're planning to use, so it's hard to specify.
|
||||
Generally you can just try using the module and then install missing packages via ~pip install --user~, should be fairly straightforward.
|
||||
|
||||
* Usage examples
|
||||
If you run your script with ~with_my~ wrapper, you'd have ~my~ in ~PYTHONPATH~ which gives you access to your data from within the script.
|
||||
|
||||
- accessing Kobo books
|
||||
|
||||
#+begin_src bash
|
||||
with_my python3 -c 'import my.books.kobo as kobo; print(kobo.get_todos())'
|
||||
#+end_src
|
||||
|
||||
- if you have [[https://github.com/karlicoss/orger][orger]] installed, you can use its modules to get Org-mode representations of your data. For instance, rendering [[https://github.com/burtonator/polar-bookshelf][Polar]] highlights as org-mode file as easy as:
|
||||
#+begin_src bash
|
||||
with_my orger/modules/polar.py --to polar.org
|
||||
#+end_src
|
||||
|
||||
- read/run [[./demo.py][demo.py]] for a full demonstration of setting up Hypothesis (it uses public annotations data from Github)
|
||||
|
||||
|
||||
* Linting
|
||||
|
||||
#+begin_src bash
|
||||
# see https://github.com/python/mypy/issues/1645 for --namespace-packages explanation
|
||||
with_my mypy --namespace-packages my
|
||||
#+end_src
|
||||
|
||||
or, set up as ~mypy.ini~ file:
|
||||
|
||||
#+begin_src
|
||||
[mypy]
|
||||
mypy_path=/path/to/mycfg_dir
|
||||
#+end_src
|
||||
Loading…
Add table
Add a link
Reference in a new issue