# TODO  FAQ??
Please don't be shy and raise issues if something in the instructions is unclear.
You'd be really helping me, I want to make the setup as straightforward as possible!

# update with org-make-toc
* TOC
:PROPERTIES:
:TOC:      :include all
:END:

:CONTENTS:
- [[#toc][TOC]]
- [[#few-notes][Few notes]]
- [[#install-main-hpi-package][Install main HPI package]]
  - [[#option-1-install-from-pip][option 1: install from PIP]]
  - [[#option-2-localeditable-install][option 2: local/editable install]]
  - [[#option-3-use-without-installing][option 3: use without installing]]
  - [[#appendix-optional-packages][appendix: optional packages]]
- [[#setting-up-modules][Setting up modules]]
  - [[#private-configuration-myconfig][private configuration (my.config)]]
  - [[#module-dependencies][module dependencies]]
- [[#troubleshooting][Troubleshooting]]
- [[#usage-examples][Usage examples]]
  - [[#end-to-end-roam-research-setup][End-to-end Roam Research setup]]
  - [[#polar][Polar]]
  - [[#google-takeout][Google Takeout]]
  - [[#kobo-reader][Kobo reader]]
  - [[#orger][Orger]]
    - [[#orger--polar][Orger + Polar]]
  - [[#demopy][demo.py]]
- [[#data-flow][Data flow]]
  - [[#polar-bookshelf][Polar Bookshelf]]
  - [[#google-takeout][Google Takeout]]
  - [[#reddit][Reddit]]
  - [[#twitter][Twitter]]
  - [[#connecting-to-other-apps][Connecting to other apps]]
- [[#addingmodifying-modules][Adding/modifying modules]]
:END:


* Few notes
I understand that people who'd like to use this may not be super familiar with Python, PIP or generally unix, so here are some useful notes:

- only ~python >= 3.6~ is supported
- I'm using ~pip3~ command, but on your system you might only have ~pip~.

  If your ~pip --version~ says python 3, feel free to use ~pip~.

- similarly, I'm using =python3= in the documentation, but if your =python --version= says python3, it's okay to use =python=

- when you are using ~pip install~, [[https://stackoverflow.com/a/42989020/706389][always pass]] =--user=, and *never install third party packages with sudo* (unless you know what you are doing)
- throughout the guide I'm assuming the user config directory is =~/.config=, but it's *different on Mac/Windows*.

  See [[https://github.com/ActiveState/appdirs/blob/3fe6a83776843a46f20c2e5587afcffe05e03b39/appdirs.py#L187-L190][this]] if you're not sure what's your user config dir.

* Install main HPI package
This is a *required step*

You can choose one of the following options:

** option 1: install from [[https://pypi.org/project/HPI][PIP]]
This is the *easiest way*:

: pip3 install --user HPI

** option 2: local/editable install
This is convenient if you're planning to add new modules or change the existing ones.

1. Clone the repository: =git clone git@github.com:karlicoss/HPI.git /path/to/hpi=
2. Go into the project directory: =cd /path/to/hpi=
2. Run  ~pip3 install --user -e .~

   This will install the package in 'editable mode'.
   It means that any changes to =/path/to/hpi= will be immediately reflected without need to reinstall anything.

   It's *extremely* convenient for developing and debugging.
  
** option 3: use without installing
This is less convenient, but gives you more control.

1. Clone the repository: =git clone git@github.com:karlicoss/HPI.git /path/to/hpi=
2. Go into the project directory: =cd /path/to/hpi=
3. Install the dependencies: ~python3 setup.py --dependencies-only~
4. Use =with_my= script to get access to ~my.~ modules.

   For example:

   : /path/to/hpi/with_my python3 -c 'from my.pinboard import bookmarks; print(list(bookmarks()))'

   It's also convenient to put a symlink to =with_my= somewhere in your system path so you can run it from anywhere, or add an alias in your bashrc:

   : alias with_my='/path/to/hpi/with_my'

   After that, you can wrap your command in =with_my= to give it access to ~my.~ modules, e.g. see [[#usage-examples][examples]].

The benefit of this way is that you get a bit more control, explicitly allowing your scripts to use your data.

** appendix: optional packages
You can also install some opional packages

: pip3 install 'HPI[optional]'

They aren't necessary, but will improve your experience. At the moment these are:

- [[https://github.com/karlicoss/cachew][cachew]]: automatic caching library, which can greatly speedup data access
- [[https://github.com/metachris/logzero][logzero]]: a nice logging library, supporting colors
- [[https://github.com/python/mypy][mypy]]: mypy is used for checking configs and troubleshooting

* Setting up modules
This is an *optional step* as few modules work without extra setup.
But it depends on the specific module.

See [[file:MODULES.org][MODULES]] to read documentation on specific modules that interest you.

You might also find interesting to read [[file:CONFIGURING.org][CONFIGURING]], where I'm
elaborating on some technical rationales behind the current configuration system.

** private configuration (=my.config=)
# TODO write about dynamic configuration
# TODO add a command to edit config?? e.g. HPI config edit
If you're not planning to use private configuration (some modules don't need it) you can skip straight to the next step. Still, I'd recommend you to read anyway.

The configuration contains paths to the data on your disks, links to external repositories, etc.
The config is simply a *python package* (named =my.config=), expected to be in =~/.config/my=.

Since it's a Python package, generally it's very *flexible* and there are many ways to set it up.

- *The simplest way*

  After installing HPI, run =hpi config init=.

  This will create an empty config file for you (usually, in =~/.config/my=), which you can edit. Example configuration:

  #+begin_src python
  import pytz # yes, you can use any Python stuff in the config

  class emfit:
      export_path = '/data/exports/emfit'
      tz = pytz.timezone('Europe/London')
      excluded_sids = []
      cache_path  = '/tmp/emfit.cache'

  class instapaper:
      export_path = '/data/exports/instapaper'

  class roamresearch:
      export_path = '/data/exports/roamresearch'
      username    = 'karlicoss'

  #+end_src

  To find out which attributes you need to specify:

  - check in [[file:MODULES.org][MODULES]]
  - if there is nothing there, the easiest is perhaps to skim through the code of the module and to search for =config.= uses.
   
    For example, if you search for =config.= in [[file:../my/emfit/__init__.py][emfit module]], you'll see that it's using =export_path=, =tz=, =excluded_sids= and =cache_path=.

  - or you can just try running them and fill in the attributes Python complains about!

- Another example is in [[file:example_config][example_config]]:

  #+begin_src bash :exports results :results output
    for x in $(find example_config/ | grep -v -E 'mypy_cache|.git|__pycache__|scignore'); do
      if   [[ -L "$x" ]]; then
        echo "symlink | $x -> $(readlink $x)"
      elif [[ -d "$x" ]]; then
        echo "dir     | $x"
      else
        echo "file    | $x"
        (echo "---"; cat "$x"; echo "---" ) | sed 's/^/          /'
      fi
    done
  #+end_src

  #+RESULTS:
  #+begin_example
  dir     | example_config/
  dir     | example_config/my
  dir     | example_config/my/config
  file    | example_config/my/config/__init__.py
            ---
            """
            Feel free to remove this if you don't need it/add your own custom settings and use them
            """

            class hypothesis:
                # expects outputs from https://github.com/karlicoss/hypexport
                # (it's just the standard Hypothes.is export format)
                export_path = '/path/to/hypothesis/data'
            ---
  dir     | example_config/my/config/repos
  symlink | example_config/my/config/repos/hypexport -> /tmp/my_demo/hypothesis_repo
  #+end_example

As you can see, generally you specify fixed paths (e.g. to your backups directory) in ~__init__.py~.
Feel free to add other files as well though to organize better, it's a real Python package after all!

Some things (e.g. links to external packages like [[https://github.com/karlicoss/hypexport][hypexport]]) are specified as *ordinary symlinks* in ~repos~ directory.
That way you get easy imports (e.g. =import my.config.repos.hypexport.model=) and proper IDE integration.

- my own config layout is a bit more complicated:

  #+begin_src python :exports results :results output
  from pathlib import Path
  home = Path("~").expanduser()
  pp = home / '.config/my/my/config'
  for p in sorted(pp.rglob('*')):
    if '__pycache__' in p.parts:
      continue
    ps = str(p).replace(str(home), '~')
    print(ps)
  #+end_src

  #+RESULTS:
  #+begin_example
  ~/.config/my/my/config/__init__.py
  ~/.config/my/my/config/locations.py
  ~/.config/my/my/config/repos
  ~/.config/my/my/config/repos/endoexport
  ~/.config/my/my/config/repos/fbmessengerexport
  ~/.config/my/my/config/repos/kobuddy
  ~/.config/my/my/config/repos/monzoexport
  ~/.config/my/my/config/repos/pockexport
  ~/.config/my/my/config/repos/rexport
  #+end_example

# TODO link to post about exports?
** module dependencies
Dependencies are different for specific modules you're planning to use, so it's hard to specify.

Generally you can just try using the module and then install missing packages via ~pip3 install --user~, should be fairly straightforward.


* Troubleshooting
# todo replace with_my with it??

HPI comes with a command line tool that can help you detect potential issues. Run:

: hpi doctor
: # alternatively, for more output:
: hpi doctor --verbose

If you only have few modules set up, lots of them will error for you, which is expected, so check the ones you expect to work.

If you have any ideas on how to improve it, please let me know!

Here's a screenshot how it looks when everything is mostly good: [[https://user-images.githubusercontent.com/291333/82806066-f7dfe400-9e7c-11ea-8763-b3bee8ada308.png][link]].

* Usage examples
If you run your script with ~with_my~ wrapper, you'd have ~my~ in ~PYTHONPATH~ which gives you access to your data from within the script.

** End-to-end Roam Research setup
In [[https://beepb00p.xyz/myinfra-roam.html#export][this]] post you can trace all steps:

- learn how to export your raw data
- integrate it with HPI package
- benefit from HPI integration

  - use interactively in ipython
  - use with [[https://github.com/karlicoss/orger][Orger]]
  - use with [[https://github.com/karlicoss/promnesia][Promnesia]]

If you want to set up a new data source, it could be a good learning reference.

** Polar
Polar doesn't require any setup as it accesses the highlights on your filesystem (usually in =~/.polar=).

You can try if it works with:

: python3 -c 'import my.reading.polar as polar; print(polar.get_entries())'

** Google Takeout
If you have zip Google Takeout archives, you can use HPI to access it:

- prepare the config =~/.config/my/my/config.py=

  #+begin_src python
  class google:
      # you can pass the directory, a glob, or a single zip file
      takeout_path = '/backups/takeouts/*.zip'
  #+end_src

- use it:

  #+begin_src
  $ python3 -c 'import my.media.youtube as yt; print(yt.get_watched()[-1])'
  Watched(url='https://www.youtube.com/watch?v=p0t0J_ERzHM', title='Monster magnet meets monster magnet...', when=datetime.datetime(2020, 1, 22, 20, 34, tzinfo=<UTC>))
  #+end_src


** Kobo reader
Kobo module allows you to access the books you've read along with the highlights and notes.
It uses exports provided by [[https://github.com/karlicoss/kobuddy][kobuddy]] package.

- prepare the config

  # todo ugh. add dynamic config...
  1. Point  =ln -sfT /path/to/kobuddy ~/.config/my/my/config/repos/kobuddy=
  2. Add kobo config to =~/.config/my/my/config/__init__.py=
    #+begin_src python
    class kobo:
        export_dir = '/backups/to/kobo/'
    #+end_src
    # TODO FIXME kobuddy path

After that you should be able to use it:

#+begin_src bash
  python3 -c 'import my.books.kobo as kobo; print(kobo.get_highlights())'
#+end_src

** Orger
# TODO include this from orger docs??

You can use [[https://github.com/karlicoss/orger][orger]] to get Org-mode representations of your data.

Some examples (assuming you've [[https://github.com/karlicoss/orger#installing][installed]] Orger):

*** Orger + [[https://github.com/burtonator/polar-bookshelf][Polar]]

This will mirror Polar highlights as org-mode:

: orger/modules/polar.py --to polar.org

** =demo.py=
read/run [[../demo.py][demo.py]] for a full demonstration of setting up Hypothesis (uses annotations data from a public Github repository)

* Data flow
# todo eh, could publish this as a blog page? dunno

Here, I'll demonstrate how data flows into and from HPI on several examples, starting from the simplest to more complicated.

If you want to see how it looks as a whole, check out [[https://beepb00p.xyz/myinfra.html#mypkg][my infrastructure map]]!

** Polar Bookshelf
Polar keeps the data:

- *locally*, on your disk
- in =~/.polar=,
- as a bunch of *JSON files*
 
It's excellent from all perspectives, except one -- you can only use meaningfully use it through Polar app.
Which is, by all means, great!

But you might want to integrate your data elsewhere and use it in ways that Polar developer never even anticipated!

If you check the data layout ([[https://github.com/TheCedarPrince/KnowledgeRepository][example]]), you can see it's messy: scattered across multiple directories, contains raw HTML, obscure entities, etc.
It's understandable from the app developer's perspective, but it makes things frustrating when you want to work with this data.

# todo hmm what if I could share deserialization with Polar app?

Here comes the HPI [[file:../my/reading/polar.py][polar module]]!

: |💾 ~/.polar (raw JSON data) |
:             ⇓⇓⇓
:    HPI (my.reading.polar)
:             ⇓⇓⇓
:    < python interface >

So the data is read from the =|💾 filesystem |=, processed/normalized with HPI, which results in a nice programmatic =< interface >= for Polar data.

Note that it doesn't require any extra configuration -- it "just" works because the data is kept locally in the *known location*.

** Google Takeout
# TODO twitter archive might be better here?
Google Takeout exports are, unfortunately, manual (or semi-manual if you do some [[https://beepb00p.xyz/my-data.html#takeout][voodoo]] with mounting Google Drive).
Anyway, say you're doing it once in six months, so you end up with a several archives on your disk:

: /backups/takeout/takeout-20151201.zip
: ....
: /backups/takeout/takeout-20190901.zip
: /backups/takeout/takeout-20200301.zip

Inside the archives.... there is a [[https://www.specytech.com/blog/wp-content/uploads/2019/06/google-takeout-folder.png][bunch]] of random files from all your google services.
Lately, many of them are JSONs, but for example, in 2015 most of it was in HTMLs! It's a nightmare to work with, even when you're an experienced programmer.

# Even within a single data source (e.g. =My Activity/Search=) you have a mix of HTML and JSON files.
# todo eh, I need to actually add JSON processing first
Of course, HPI helps you here by encapsulating all this parsing logic and exposing Python interfaces instead.

:       < 🌐  Google |
:              ⇓⇓⇓
:     { manual download }
:              ⇓⇓⇓
:  |💾 /backups/takeout/*.zip |
:              ⇓⇓⇓
:    HPI (my.google.takeout)
:              ⇓⇓⇓
:     < python interface >

The only thing you need to do is to tell it where to find the files on your disk, via [[file:MODULES.org::#mygoogletakeoutpaths][the config]], because different people use different paths for backups.

# TODO how to emphasize config?
# TODO python is just one of the interfaces?

** Reddit

Reddit has a proper API, so in theory HPI could talk directly to Reddit and retrieve the latest data. But that's not what it doing!

- first, there are excellent programmatic APIs for Reddit out there already, for example, [[https://github.com/praw-dev/praw][praw]]
- more importantly, this is the [[https://beepb00p.xyz/exports.html#design][design decision]] of HP

  It doesn't deal with all with the complexities of API interactions.
  Instead, it relies on other tools to put *intermediate, raw data*, on your disk and then transforms this data into something nice.

As an example, for [[file:../my/reddit.py][Reddit]], HPI is relying on data fetched by [[https://github.com/karlicoss/rexport][rexport]] library. So the pipeline looks like:

:       < 🌐  Reddit |
:              ⇓⇓⇓
:     { rexport/export.py (automatic, e.g. cron) }
:              ⇓⇓⇓
:  |💾 /backups/reddit/*.json |
:              ⇓⇓⇓
:      HPI (my.reddit)
:              ⇓⇓⇓
:     < python interface >

So, in your [[file:MODULES.org::#myreddit][reddit config]], similarly to Takeout, you need =export_path=, so HPI knows how to find your Reddit data on the disk.

But there is an extra caveat: rexport is already coming with nice [[https://github.com/karlicoss/rexport/blob/master/dal.py][data bindings]] to parse its outputs.
Another *design decision* of HPI is to use existing code and libraries as much as possible, so we also specify a path to =rexport= repository in the config.

(note: in the future it's possible that rexport will be installed via PIP, I just haven't had time for it so far).

Several other HPI modules are following a similar pattern: hypothesis, instapaper, pinboard, kobo, etc.

** Twitter

Twitter is interesting, because it's an example of a data source that *arbitrates* between several data sources from the same service.

The reason to use multiple in case of Twitter is:

- there is official Twitter Archive, but it's manual, takes several days to complete and hard to automate.
- there is [[https://github.com/twintproject/twint][twint]], which can get real-time Twitter data via scraping

  But Twitter has a limitation and you can't get data past 3200 tweets through API or scraping.

So the idea is to export both data sources on your disk:

:                              < 🌐  Twitter |
:                              ⇓⇓            ⇓⇓
:     { manual archive download }           { twint (automatic, cron) }
:              ⇓⇓⇓                                   ⇓⇓⇓
:  |💾 /backups/twitter-archives/*.zip |     |💾 /backups/twint/db.sqlite |
:                                 .............

# TODO note that the left and right parts of the diagram ('before filesystem' and 'after filesystem') are completely independent!
# if something breaks, you can still read your old data from the filesystem!

What we do next is:

1. Process raw data from twitter archives (manual export, but has all the data)
2. Process raw data from twint database (automatic export, but only recent data)
3. Merge them together, overlaying twint data on top of twitter archive data

:                                 .............
:  |💾 /backups/twitter-archives/*.zip |     |💾 /backups/twint/db.sqlite |
:              ⇓⇓⇓                                   ⇓⇓⇓
:      HPI (my.twitter.archive)              HPI (my.twitter.twint)
:       ⇓                     ⇓              ⇓                    ⇓
:       ⇓                   HPI (my.twitter.all)                  ⇓
:       ⇓                           ⇓⇓                            ⇓
: < python interface>       < python interface>          < python interface>

For merging the data, we're using a tiny auxiliary module, =my.twitter.all= (It's just 20 lines of code, [[file:../my/twitter/all.py][check it out]]).

Since you have two different sources of raw data, you need to specify two bits of config:
# todo link to modules thing?

: class twint:
:     export_path = '/backups/twint/db.sqlite'

: class twitter_archive:
:     export_path = '/backups/twitter-archives/*.zip'

Note that you can also just use =my.twitter.archive= or =my.twitter.twint= directly, or set either of paths to 'empty path': =()=
# TODO empty string?
# (TODO mypy-safe?)

# #addingmodifying-modules
# Now, say you prefer to use a different library for your Twitter data instead of twint (for whatever reason), and you want to use it TODO
# TODO docs on overlays?

** Connecting to other apps
As a user you might not be so interested in Python interface per se.. but a nice thing about having one is that it's easy to
connect the data with other apps and libraries!

:                          /---- 💻promnesia --- | browser extension  >
: | python interface > ----+---- 💻orger     --- |💾 org-mode mirror  |
:                          +-----💻memacs    --- |💾 org-mode lifelog |
:                          +-----💻????      --- | REST api           >
:                          +-----💻????      --- | Datasette          >
:                          \-----💻????      --- | Memex              >

See more in [[file:../README.org::#how-do-you-use-it]["How do you use it?"]] section.

# TODO memacs module would be nice
# todo dashboard?
# todo more examples?

* Adding/modifying modules
# TODO link to 'overlays' documentation?
# TODO don't be afraid to TODO make sure to install in editable mode

The easiest is just to run HPI via [[#use-without-installing][with_my]] wrapper or with an editable PIP install.
That way your changes will be reflected immediately, and you will be able to quickly iterate/fix bugs/add new methods.

# TODO eh. doesn't even have to be in 'my' namespace?? need to check it
The "proper way" (unless you want to contribute to the upstream) is to create a separate file hierarchy and add your module to =PYTHONPATH=.

For example, if you want to add an =awesomedatasource=, it could be:

: custom_module
: └── my
:     └──awesomedatasource.py

You can use all existing HPI modules in =awesomedatasource.py=, for example, =my.config=, or everything from =my.core=.

But also, you can use *override* the builtin HPI modules too:

: custom_reddit_overlay
: └── my
:     └──reddit.py

# TODO confusing
Now if you add =my_reddit_overlay= *in the front* of ~PYTHONPATH~, all the downstream scripts using =my.reddit= will load it from =custom_reddit_overlay= instead.

This could be useful to monkey patch some behaviours, or dynamically add some extra data sources -- anything that comes to your mind.

I'll put up a better guide on this, in the meantime see [[https://packaging.python.org/guides/packaging-namespace-packages]["namespace packages"]] for more info.

# TODO add example with overriding 'all'