+
+* How do you use it?
+Mainly I use it as a data provider for my scripts, tools, and dashboards.
+
+Also, check out [[https://beepb00p.xyz/myinfra.html#mypkg][my infrastructure map]].
+It's a draft at the moment, but it might be helpful for understanding what's my vision on HPI.
+** Instant search
+Typical search interfaces make me unhappy as they are *siloed, slow, awkward to use and don't work offline*.
+So I built my own ways around it! I write about it in detail [[https://beepb00p.xyz/pkm-search.html#personal_information][here]].
+
+In essence, I'm mirroring most of my online data like chat logs, comments, etc., as plaintext.
+I can overview it in any text editor, and incrementally search over *all of it* in a single keypress.
+** orger
+[[https://github.com/karlicoss/orger][orger]] is a tool and set of modules for accessing data via org-mode.
+It allows searching and overviewing, and in addition, I'm using it for creating tasks straight from native app interfaces (e.g. Reddit/Telegram) and spaced repetition via [[https://orgmode.org/worg/org-contrib/org-drill.html][org-drill]].
+
+I write about it in detail [[https://beepb00p.xyz/orger.html][here]] and [[https://beepb00p.xyz/orger-todos.html][here]].
+** promnesia
+[[https://github.com/karlicoss/promnesia#demo][promnesia]] is a browser extension I'm working on to escape silos by *unifying annotations and browsing history* from different data sources.
+
+I've been using it for more than a year now and working on final touches to properly release it for other people.
+** dashboard
+As a big fan of [[https://beepb00p.xyz/tags.html#quantified-self][#quantified-self]], I'm working on personal health, sleep and exercise dashboard, built from various data sources.
+
+I'm working on making it public, you can see some screenshots [[https://www.reddit.com/r/QuantifiedSelf/comments/cokt4f/what_do_you_all_do_with_your_data/ewmucgk][here]].
+** timeline
+Timeline is a [[https://beepb00p.xyz/tags.html#lifelogging][#lifelogging]] project I'm working on.
+
+I want to see all my digital history, search in it, filter, easily jump at a specific point in time and see the context when it happened.
+That way it works as a sort of external memory.
+
+Ideally, it would look similar to Andrew Louis's [[https://hyfen.net/memex][Memex]], or might even reuse his interface if
+he open sources it. I highly recommend watching his talk for inspiration.
+
+* Ad-hoc and interactive
+
+
+** What were my music listening stats for 2018?
+Single import away from getting tracks you listened to:
+
+#+begin_src python
+ from my.lastfm import get_scrobbles
+ scrobbles = get_scrobbles()
+ scrobbles[200: 205]
+#+end_src
+
+
+: [Scrobble(raw={'album': 'Nevermind', 'artist': 'Nirvana', 'date': '1282488504', 'name': 'Drain You'}),
+: Scrobble(raw={'album': 'Dirt', 'artist': 'Alice in Chains', 'date': '1282489764', 'name': 'Would?'}),
+: Scrobble(raw={'album': 'Bob Dylan: The Collection', 'artist': 'Bob Dylan', 'date': '1282493517', 'name': 'Like a Rolling Stone'}),
+: Scrobble(raw={'album': 'Dark Passion Play', 'artist': 'Nightwish', 'date': '1282493819', 'name': 'Amaranth'}),
+: Scrobble(raw={'album': 'Rolled Gold +', 'artist': 'The Rolling Stones', 'date': '1282494161', 'name': "You Can't Always Get What You Want"})]
+
+
+Or, as a pandas frame to make it pretty:
+
+#+begin_src python
+ import pandas as pd
+ df = pd.DataFrame([{
+ 'dt': s.dt,
+ 'track': s.track,
+ } for s in scrobbles])
+ cdf = df.set_index('dt')
+ cdf[200: 205]
+#+end_src
+
+
+: track
+: dt
+: 2010-08-22 14:48:24+00:00 Nirvana — Drain You
+: 2010-08-22 15:09:24+00:00 Alice in Chains — Would?
+: 2010-08-22 16:11:57+00:00 Bob Dylan — Like a Rolling Stone
+: 2010-08-22 16:16:59+00:00 Nightwish — Amaranth
+: 2010-08-22 16:22:41+00:00 The Rolling Stones — You Can't Always Get What...
+
+
+We can use [[https://github.com/martijnvermaat/calmap][calmap]] library to plot a github-style music listening activity heatmap:
+
+#+begin_src python
+ import matplotlib.pyplot as plt
+ plt.figure(figsize=(10, 2.3))
+
+ import calmap
+ cdf = cdf.set_index(cdf.index.tz_localize(None)) # calmap expects tz-unaware dates
+ calmap.yearplot(cdf['track'], how='count', year=2018)
+
+ plt.tight_layout()
+ plt.title('My music listening activity for 2018')
+ plot_file = 'lastfm_2018.png'
+ plt.savefig(plot_file)
+ plot_file
+#+end_src
+
+
+[[https://beepb00p.xyz/lastfm_2018.png]]
+
+This isn't necessarily very insightful data, but fun to look at now and then!
+
+** What are the most interesting Slate Star Codex posts I've read?
+My friend asked me if I could recommend them posts I found interesting on [[https://slatestarcodex.com][Slate Star Codex]].
+With few lines of Python I can quickly recommend them posts I engaged most with, i.e. the ones I annotated most on [[https://hypothes.is][Hypothesis]].
+
+#+begin_src python
+ from my.hypothesis import get_pages
+ from collections import Counter
+ cc = Counter({p.url: len(p.highlights) for p in get_pages() if 'slatestarcodex' in p.url})
+ return cc.most_common(10)
+#+end_src
+
+
+| http://slatestarcodex.com/2013/10/20/the-anti-reactionary-faq/ | 32 |
+| https://slatestarcodex.com/2013/03/03/reactionary-philosophy-in-an-enormous-planet-sized-nutshell/ | 17 |
+| http://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/ | 16 |
+| https://slatestarcodex.com/2014/03/17/what-universal-human-experiences-are-you-missing-without-realizing-it/ | 16 |
+| http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ | 12 |
+| http://slatestarcodex.com/2015/04/21/universal-love-said-the-cactus-person/ | 11 |
+| http://slatestarcodex.com/2015/01/01/untitled/ | 11 |
+| https://slatestarcodex.com/2017/02/09/considerations-on-cost-disease/ | 10 |
+| http://slatestarcodex.com/2013/04/25/in-defense-of-psych-treatment-for-attempted-suicide/ | 9 |
+| https://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/ | 9 |
+
+** Accessing exercise data
+ E.g. see use of ~my.workouts~ [[https://beepb00p.xyz/./heartbeats_vs_kcals.html][here]].
+
+** Book reading progress
+I publish my reading stats on [[https://www.goodreads.com/user/show/22191391-dima-gerasimov][Goodreads]] so other people can see what I'm reading/have read, but Kobo [[https://beepb00p.xyz/ideas.html#kobo2goodreads][lacks integration]] with Goodreads.
+I'm using [[https://github.com/karlicoss/kobuddy][kobuddy]] to access my my Kobo data, and I've got a regular task that reminds me to sync my progress once a month.
+
+The task looks like this:
+
+#+begin_src org
+ ,* TODO [#C] sync [[https://goodreads.com][reading progress]] with kobo
+ DEADLINE: <2019-11-24 Sun .+4w -0d>
+ [[eshell: with_my python3 -c 'import my.books.kobo as kobo; kobo.print_progress()']]
+#+end_src
+
+With a single Enter keypress on the inlined =eshell:= command I can print the progress and fill in the completed books on Goodreads, e.g.:
+
+
+#+begin_example
+
+ A_Mathematician's_Apology by G. H. Hardy
+ Started : 21 Aug 2018 11:44
+ Finished: 22 Aug 2018 12:32
+
+ Fear and Loathing in Las Vegas: A Savage Journey to the Heart of the American Dream (Vintage) by Thompson, Hunter S.
+ Started : 06 Sep 2018 05:54
+ Finished: 09 Sep 2018 12:21
+
+ Sapiens: A Brief History of Humankind by Yuval Noah Harari
+ Started : 09 Sep 2018 12:22
+ Finished: 16 Sep 2018 07:25
+
+ Inadequate Equilibria: Where and How Civilizations Get Stuck by Eliezer Yudkowsky
+ Started : 31 Jul 2018 22:54
+ Finished: 16 Sep 2018 07:25
+
+ Albion Dreaming by Andy Roberts
+ Started : 20 Aug 2018 21:16
+ Finished: 16 Sep 2018 07:26
+#+end_example
+
+** Messenger stats
+How much do I chat on Facebook Messenger?
+
+#+begin_src python
+ from my.fbmessenger import messages
+
+ import pandas as pd
+ import matplotlib.pyplot as plt
+
+ df = pd.DataFrame({'dt': m.dt, 'messages': 1} for m in messages())
+ df.set_index('dt', inplace=True)
+
+ df = df.resample('M').sum() # by month
+ df = df.loc['2016-01-01':'2019-01-01'] # past subset for determinism
+
+ fig, ax = plt.subplots(figsize=(15, 5))
+ df.plot(kind='bar', ax=ax)
+
+ # todo wonder if that vvv can be less verbose...
+ x_labels = df.index.strftime('%Y %b')
+ ax.set_xticklabels(x_labels)
+
+ plot_file = 'messenger_2016_to_2019.png'
+ plt.tight_layout()
+ plt.savefig(plot_file)
+ return plot_file
+#+end_src
+
+
+[[https://beepb00p.xyz/messenger_2016_to_2019.png]]
+
+
+* How does it get input data?
+If you're curious about any specific data sources I'm using, I've written it up [[https://beepb00p.xyz/my-data.html][in detail]].
+
+In short:
+
+- The data is [[https://beepb00p.xyz/myinfra.html#exports][periodically synchronized]] from the services (cloud or not) locally, on the filesystem
+
+ As a result, you get [[https://beepb00p.xyz/myinfra.html#fs][JSONs/sqlite]] (or other formats, depending on the service) on your disk.
+
+ Once you have it, it's trivial to back it up and synchronize to other computers/phones, if necessary.
+
+ To schedule periodic sync, I'm using [[https://beepb00p.xyz/scheduler.html#cron][cron]].
+
+- =my.= package only accesses the data on the filesystem
+
+ That makes it extremely fast, reliable, and fully offline capable.
+
+As you can see, in such a setup, the data is lagging behind the 'realtime'.
+I consider it a necessary sacrifice to make everything fast and resilient.
+
+In theory, it's possible to make the system almost realtime by having a service that sucks in data continuously (rather than periodically), but it's harder as well.
+
+* Q & A
+
+** Why Python?
+I don't consider Python unique as a language suitable for such a project.
+It just happens to be the one I'm most comfortable with.
+I do have some reasons that I think make it /specifically/ good, but explaining them is out of this post's scope.
+
+In addition, Python offers a [[https://github.com/karlicoss/awesome-python#data-analysis][very rich ecosystem]] for data analysis, which we can use to our benefit.
+
+That said, I've never seen anything similar in other programming languages, and I would be really interested in, so please send me links if you know some.
+I've heard LISPs are great for data? ;)
+
+Overall, I wish [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFIs]] were a bit more mature, so we didn't have to think about specific programming languages at all.
+
+** Can anyone use it?
+Yes!
+
+- you can plug in your own data
+- most modules are isolated, so you can only use the ones that you want to
+- everything is easily extensible
+
+ Starting from simply adding new modules to any dynamic hackery you can possibly imagine within Python.
+
+** How easy is it to use?
+The whole setup requires some basic programmer literacy:
+
+- installing/running and potentially modifying Python code
+- using symlinks
+- potentially running Cron jobs
+
+If you have any ideas on making the setup simpler, please let me know!
+
+** What about privacy?
+The modules contain no data, only code to operate on the data.
+
+Everything is [[https://beepb00p.xyz/tags.html#offline][local fist]], the input data is on your filesystem.
+If you're truly paranoid, you can even wrap it in a Docker container.
+
+There is still a question of whether you trust yourself at even keeping all the data on your disk, but it is out of the scope of this post.
+
+If you'd rather keep some code private too, it's also trivial to achieve with a private subpackage.
+
+** But /should/ I use it?
+#+begin_quote
+Sure, maybe you can achieve a perfect system where you can instantly find and recall anything that you've done. Do you really want it?
+Wouldn't that, like, make you less human?
+#+end_quote
+
+I'm not a gatekeeper of what it means to be human, but I don't think that the shortcomings of the human brain are what makes us such.
+
+So I can't answer that for you. I certainly want it though.
+I'm [[https://beepb00p.xyz/tags.html#pkm][quite open]] about my goals -- I'd happily get merged/augmented with a computer to enhance my thinking and analytical abilities.
+
+While at the moment [[https://en.wikipedia.org/wiki/Hard_problem_of_consciousness][we don't even remotely understand]] what would such merging or "mind uploading" entail exactly,
+I can clearly delegate some tasks, like long term memory, information lookup, and data processing to a computer. They can already handle it really well.
+
+#+begin_quote
+What about these people who have perfect recall and wish they hadn't.
+#+end_quote
+
+Sure, maybe it sucks. At the moment though, I don't anything close to it and this only annoys me.
+I want to have a choice at least, and digital tools give me this choice.
+
+** Would it suit /me/?
+Probably, at least to some extent.
+
+First, our lives are different, so our APIs might be different too.
+This is more of a demonstration of what's I'm using, although I did spend effort towards making it as modular and extensible as possible, so other people could use it too.
+It's easy to modify code, add extra methods and modules. You can even keep all your modifications private.
+
+But after all, we've all sharing many similar activities and using the same products, so there is a huge overlap.
+I'm not sure how far we can stretch it and keep modules generic enough to be used by multiple people. But let's give it a try perhaps? :)
+
+Second, interacting with your data through the code is the central idea of the project.
+That kind of cuts off people without technical skills, and even many people capable of coding,
+who dislike the idea of writing code outside of work.
+
+It might be possible to expose some [[https://en.wikipedia.org/wiki/No-code_development_platform][no-code]] interfaces,
+but I still feel that wouldn't be enough.
+
+I'm not sure whether it's a solvable problem at this point, but happy to hear any suggestions!
+
+** What it isn't?
+- It's not vaporwave
+
+ The project is a little crude, but it's real and working. I've been using it for a long time now, and find it fairly sustainable to keep using for the foreseeable future.
+
+- It's not going to be another silo
+
+ While I don't have anything against commercial use (and I believe any work in this area will benefit all of us), I'm not planning to build a product out of it.
+
+ I really hope it can grow into or inspire some mature open source system.
+
+ Please take my ideas and code and build something cool from it!
+
+
+* Related links
+Similar projects:
+
+- [[https://github.com/novoid/Memacs][Memacs]] by Karl Voit
+- [[https://news.ycombinator.com/item?id=9615901][Me API - turn yourself into an open API (HN)]]
+- [[https://github.com/markwk/qs_ledger][QS ledger]] from Mark Koester
+
+- [[https://github.com/tehmantra/my][tehmantra/my]]: directly inspired by this package
+- [[https://github.com/bcongdon/bolero][bcongdon/bolero]]
+- [[https://en.wikipedia.org/wiki/Solid_(web_decentralization_project)#Design][Solid project]]: personal data pod, which websites pull data from
+
+Other links:
+
+- NetOpWibby: [[https://news.ycombinator.com/item?id=21684949][A Personal API (HN)]]
+- [[https://beepb00p.xyz/sad-infra.html][The sad state of personal data and infrastructure]]: here I am going into motivation and difficulties arising in the implementation
+
+* --
+Open to any feedback and thoughts!
+
+Also, don't hesitate to raise an issue, or reach me personally if you want to try using it, and find the instructions confusing. Your questions would help me to make it simpler!
+
+In some near future I will write more about:
+
+- specific technical decisions and patterns
+- challenges I had so solve
+- more use-cases and demos -- it's impossible to fit everything in one post!
+
+, but happy to answer any questions on these topics now!
\ No newline at end of file
diff --git a/doc/SETUP.org b/doc/SETUP.org
index 6bfce94..3560e2a 100644
--- a/doc/SETUP.org
+++ b/doc/SETUP.org
@@ -1,7 +1,7 @@
-# TODO hmm, preamble?
-# TODO hci??
-[[https://circleci.com/gh/karlicoss/my/tree/master][https://circleci.com/gh/karlicoss/my/tree/master.svg?style=svg]]
+# [[https://circleci.com/gh/karlicoss/my/tree/master][https://circleci.com/gh/karlicoss/my/tree/master.svg?style=svg]]
+Please don't be shy and raise issues if something in the instructions is unclear.
+You'd be really helping me, I want to make the setup as straightforward as possible!
* Setting up
** =mycfg= package for private paths/repositories (optional)