fz0x1/HPI
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore: update urls

Browse source
This commit is contained in:
purarue 2024-10-25 09:39:00 -07:00 • committed by karlicoss
parent a2b397ec4a
commit 7ab6f0d5cb
18 changed files with 38 additions and 38 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
README.org
View file

@ -723,10 +723,10 @@ If you want to write modules for personal use but don't want to merge them into
Other HPI Repositories: Other HPI Repositories:
- [[https://github.com/seanbreckenridge/HPI][seanbreckenridge/HPI]] - [[https://github.com/purarue/HPI][purarue/HPI]]
- [[https://github.com/madelinecameron/hpi][madelinecameron/HPI]] - [[https://github.com/madelinecameron/hpi][madelinecameron/HPI]]
If you want to create your own to create your own modules/override something here, you can use the [[https://github.com/seanbreckenridge/HPI-template][template]]. If you want to create your own to create your own modules/override something here, you can use the [[https://github.com/purarue/HPI-template][template]].
* Related links * Related links
:PROPERTIES: :PROPERTIES:

6
doc/DENYLIST.md
View file

@ -76,7 +76,7 @@ This would typically be used in an overridden `all.py` file, or in a one-off scr
which you may want to filter out some items from a source, progressively adding more which you may want to filter out some items from a source, progressively adding more
items to the denylist as you go. items to the denylist as you go.
A potential `my/ip/all.py` file might look like (Sidenote: `discord` module from [here](https://github.com/seanbreckenridge/HPI)): A potential `my/ip/all.py` file might look like (Sidenote: `discord` module from [here](https://github.com/purarue/HPI)):
```python ```python
from typing import Iterator from typing import Iterator
@ -119,9 +119,9 @@ python3 -c 'from my.ip import all; all.deny.deny_cli(all.ips())'
To edit the `all.py`, you could either: To edit the `all.py`, you could either:
- install it as editable (`python3 -m pip install --user -e ./HPI`), and then edit the file directly - install it as editable (`python3 -m pip install --user -e ./HPI`), and then edit the file directly
- or, create a namespace package, which splits the package across multiple directories. For info on that see [`MODULE_DESIGN`](https://github.com/karlicoss/HPI/blob/master/doc/MODULE_DESIGN.org#namespace-packages), [`reorder_editable`](https://github.com/seanbreckenridge/reorder_editable), and possibly the [`HPI-template`](https://github.com/seanbreckenridge/HPI-template) to create your own HPI namespace package to create your own `all.py` file. - or, create a namespace package, which splits the package across multiple directories. For info on that see [`MODULE_DESIGN`](https://github.com/karlicoss/HPI/blob/master/doc/MODULE_DESIGN.org#namespace-packages), [`reorder_editable`](https://github.com/purarue/reorder_editable), and possibly the [`HPI-template`](https://github.com/purarue/HPI-template) to create your own HPI namespace package to create your own `all.py` file.
For a real example of this see, [seanbreckenridge/HPI-personal](https://github.com/seanbreckenridge/HPI-personal/blob/master/my/ip/all.py) For a real example of this see, [purarue/HPI-personal](https://github.com/purarue/HPI-personal/blob/master/my/ip/all.py)
Sidenote: the reason why we want to specifically override Sidenote: the reason why we want to specifically override
the all.py and not just create a script that filters out the items you're the all.py and not just create a script that filters out the items you're

12
doc/MODULES.org
View file

@ -76,7 +76,7 @@ The config snippets below are meant to be modified accordingly and *pasted into
You don't have to set up all modules at once, it's recommended to do it gradually, to get the feel of how HPI works. You don't have to set up all modules at once, it's recommended to do it gradually, to get the feel of how HPI works.
For an extensive/complex example, you can check out ~@seanbreckenridge~'s [[https://github.com/seanbreckenridge/dotfiles/blob/master/.config/my/my/config/__init__.py][config]] For an extensive/complex example, you can check out ~@purarue~'s [[https://github.com/purarue/dotfiles/blob/master/.config/my/my/config/__init__.py][config]]
# Nested Configurations before the doc generation using the block below # Nested Configurations before the doc generation using the block below
** [[file:../my/reddit][my.reddit]] ** [[file:../my/reddit][my.reddit]]
@ -96,7 +96,7 @@ For an extensive/complex example, you can check out ~@seanbreckenridge~'s [[http
class pushshift: class pushshift:
''' '''
Uses [[https://github.com/seanbreckenridge/pushshift_comment_export][pushshift]] to get access to old comments Uses [[https://github.com/purarue/pushshift_comment_export][pushshift]] to get access to old comments
''' '''
# path[s]/glob to the exported JSON data # path[s]/glob to the exported JSON data
@ -106,7 +106,7 @@ For an extensive/complex example, you can check out ~@seanbreckenridge~'s [[http
** [[file:../my/browser/][my.browser]] ** [[file:../my/browser/][my.browser]]
Parses browser history using [[http://github.com/seanbreckenridge/browserexport][browserexport]] Parses browser history using [[http://github.com/purarue/browserexport][browserexport]]
#+begin_src python #+begin_src python
class browser: class browser:
@ -132,7 +132,7 @@ For an extensive/complex example, you can check out ~@seanbreckenridge~'s [[http
You might also be able to use [[file:../my/location/via_ip.py][my.location.via_ip]] which uses =my.ip.all= to You might also be able to use [[file:../my/location/via_ip.py][my.location.via_ip]] which uses =my.ip.all= to
provide geolocation data for an IPs (though no IPs are provided from any provide geolocation data for an IPs (though no IPs are provided from any
of the sources here). For an example of usage, see [[https://github.com/seanbreckenridge/HPI/tree/master/my/ip][here]] of the sources here). For an example of usage, see [[https://github.com/purarue/HPI/tree/master/my/ip][here]]
#+begin_src python #+begin_src python
class location: class location:
@ -256,9 +256,9 @@ for cls, p in modules:
** [[file:../my/google/takeout/parser.py][my.google.takeout.parser]] ** [[file:../my/google/takeout/parser.py][my.google.takeout.parser]]
Parses Google Takeout using [[https://github.com/seanbreckenridge/google_takeout_parser][google_takeout_parser]] Parses Google Takeout using [[https://github.com/purarue/google_takeout_parser][google_takeout_parser]]
See [[https://github.com/seanbreckenridge/google_takeout_parser][google_takeout_parser]] for more information about how to export and organize your takeouts See [[https://github.com/purarue/google_takeout_parser][google_takeout_parser]] for more information about how to export and organize your takeouts
If the =DISABLE_TAKEOUT_CACHE= environment variable is set, this won't If the =DISABLE_TAKEOUT_CACHE= environment variable is set, this won't
cache individual exports in =~/.cache/google_takeout_parser= cache individual exports in =~/.cache/google_takeout_parser=

8
doc/MODULE_DESIGN.org
View file

@ -67,7 +67,7 @@ If you want to disable a source, you have a few options.
... that suppresses the warning message and lets you use ~my.location.all~ without having to change any lines of code ... that suppresses the warning message and lets you use ~my.location.all~ without having to change any lines of code
Another benefit is that all the custom sources/data is localized to the ~all.py~ file, so a user can override the ~all.py~ (see the sections below on ~namespace packages~) file in their own HPI repository, adding additional sources without having to maintain a fork and patching in changes as things eventually change. For a 'real world' example of that, see [[https://github.com/seanbreckenridge/HPI#partially-in-usewith-overrides][seanbreckenridge]]s location and ip modules. Another benefit is that all the custom sources/data is localized to the ~all.py~ file, so a user can override the ~all.py~ (see the sections below on ~namespace packages~) file in their own HPI repository, adding additional sources without having to maintain a fork and patching in changes as things eventually change. For a 'real world' example of that, see [[https://github.com/purarue/HPI#partially-in-usewith-overrides][purarue]]s location and ip modules.
This is of course not required for personal or single file modules, its just the pattern that seems to have the least amount of friction for the user, while being extendable, and without using a bulky plugin system to let users add additional sources. This is of course not required for personal or single file modules, its just the pattern that seems to have the least amount of friction for the user, while being extendable, and without using a bulky plugin system to let users add additional sources.
@ -208,13 +208,13 @@ Where ~lastfm.py~ is your version of ~my.lastfm~, which you've copied from this
Then, running ~python3 -m pip install -e .~ in that directory would install that as part of the namespace package, and assuming (see below for possible issues) this appears on ~sys.path~ before the upstream repository, your ~lastfm.py~ file overrides the upstream. Adding more files, like ~my.some_new_module~ into that directory immediately updates the global ~my~ package -- allowing you to quickly add new modules without having to re-install. Then, running ~python3 -m pip install -e .~ in that directory would install that as part of the namespace package, and assuming (see below for possible issues) this appears on ~sys.path~ before the upstream repository, your ~lastfm.py~ file overrides the upstream. Adding more files, like ~my.some_new_module~ into that directory immediately updates the global ~my~ package -- allowing you to quickly add new modules without having to re-install.
If you install both directories as editable packages (which has the benefit of any changes you making in either repository immediately updating the globally installed ~my~ package), there are some concerns with which editable install appears on your ~sys.path~ first. If you wanted your modules to override the upstream modules, yours would have to appear on the ~sys.path~ first (this is the same reason that =custom_lastfm_overlay= must be at the front of your ~PYTHONPATH~). For more details and examples on dealing with editable namespace packages in the context of HPI, see the [[https://github.com/seanbreckenridge/reorder_editable][reorder_editable]] repository. If you install both directories as editable packages (which has the benefit of any changes you making in either repository immediately updating the globally installed ~my~ package), there are some concerns with which editable install appears on your ~sys.path~ first. If you wanted your modules to override the upstream modules, yours would have to appear on the ~sys.path~ first (this is the same reason that =custom_lastfm_overlay= must be at the front of your ~PYTHONPATH~). For more details and examples on dealing with editable namespace packages in the context of HPI, see the [[https://github.com/purarue/reorder_editable][reorder_editable]] repository.
There is no limit to how many directories you could install into a single namespace package, which could be a possible way for people to install additional HPI modules, without worrying about the module count here becoming too large to manage. There is no limit to how many directories you could install into a single namespace package, which could be a possible way for people to install additional HPI modules, without worrying about the module count here becoming too large to manage.
There are some other users [[https://github.com/hpi/hpi][who have begun publishing their own modules]] as namespace packages, which you could potentially install and use, in addition to this repository, if any of those interest you. If you want to create your own you can use the [[https://github.com/seanbreckenridge/HPI-template][template]] to get started. There are some other users [[https://github.com/hpi/hpi][who have begun publishing their own modules]] as namespace packages, which you could potentially install and use, in addition to this repository, if any of those interest you. If you want to create your own you can use the [[https://github.com/purarue/HPI-template][template]] to get started.
Though, enabling this many modules may make ~hpi doctor~ look pretty busy. You can explicitly choose to enable/disable modules with a list of modules/regexes in your [[https://github.com/karlicoss/HPI/blob/f559e7cb899107538e6c6bbcf7576780604697ef/my/core/core_config.py#L24-L55][core config]], see [[https://github.com/seanbreckenridge/dotfiles/blob/a1a77c581de31bd55a6af3d11b8af588614a207e/.config/my/my/config/__init__.py#L42-L72][here]] for an example. Though, enabling this many modules may make ~hpi doctor~ look pretty busy. You can explicitly choose to enable/disable modules with a list of modules/regexes in your [[https://github.com/karlicoss/HPI/blob/f559e7cb899107538e6c6bbcf7576780604697ef/my/core/core_config.py#L24-L55][core config]], see [[https://github.com/purarue/dotfiles/blob/a1a77c581de31bd55a6af3d11b8af588614a207e/.config/my/my/config/__init__.py#L42-L72][here]] for an example.
You may use the other modules or [[https://github.com/karlicoss/hpi-personal-overlay][my overlay]] as reference, but python packaging is already a complicated issue, before adding complexities like namespace packages and editable installs on top of it... If you're having trouble extending HPI in this fashion, you can open an issue here, preferably with a link to your code/repository and/or ~setup.py~ you're trying to use. You may use the other modules or [[https://github.com/karlicoss/hpi-personal-overlay][my overlay]] as reference, but python packaging is already a complicated issue, before adding complexities like namespace packages and editable installs on top of it... If you're having trouble extending HPI in this fashion, you can open an issue here, preferably with a link to your code/repository and/or ~setup.py~ you're trying to use.

2
doc/OVERLAYS.org
View file

@ -66,7 +66,7 @@ This basically means that modules will be searched in both paths, with overlay t
** Installing with =--use-pep517= ** Installing with =--use-pep517=
See here for discussion https://github.com/seanbreckenridge/reorder_editable/issues/2, but TLDR it should work similarly. See here for discussion https://github.com/purarue/reorder_editable/issues/2, but TLDR it should work similarly.
* Testing runtime behaviour (editable install) * Testing runtime behaviour (editable install)

6
doc/QUERY.md
View file

@ -99,7 +99,7 @@ Commit(committed_dt=datetime.datetime(2023, 4, 14, 23, 9, 1, tzinfo=datetime.tim
authored_dt=datetime.datetime(2023, 4, 14, 23, 4, 1, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=61200))), authored_dt=datetime.datetime(2023, 4, 14, 23, 4, 1, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=61200))),
message='sources.smscalls: propogate errors if there are breaking ' message='sources.smscalls: propogate errors if there are breaking '
'schema changes', 'schema changes',
repo='/home/sean/Repos/promnesia-fork', repo='/home/username/Repos/promnesia-fork',
sha='22a434fca9a28df9b0915ccf16368df129d2c9ce', sha='22a434fca9a28df9b0915ccf16368df129d2c9ce',
ref='refs/heads/smscalls-handle-result') ref='refs/heads/smscalls-handle-result')
``` ```
@ -195,7 +195,7 @@ To preview, you can use something like [`qgis`](https://qgis.org/en/site/) or fo
<img src="https://user-images.githubusercontent.com/7804791/232249184-7e203ee6-a3ec-4053-800c-751d2c28e690.png" width=500 alt="chicago trip" /> <img src="https://user-images.githubusercontent.com/7804791/232249184-7e203ee6-a3ec-4053-800c-751d2c28e690.png" width=500 alt="chicago trip" />
(Sidenote: this is [`@seanbreckenridge`](https://github.com/seanbreckenridge/)s locations, on a trip to Chicago) (Sidenote: this is [`@purarue`](https://github.com/purarue/)s locations, on a trip to Chicago)
## Python reference ## Python reference
@ -301,4 +301,4 @@ The `hpi query` command is a CLI wrapper around the code in [`query.py`](../my/c
If you specify a range, drop_unsorted is forced to be True If you specify a range, drop_unsorted is forced to be True
``` ```
Those can be imported and accept any sort of iterator, `hpi query` just defaults to the output of functions here. As an example, see [`listens`](https://github.com/seanbreckenridge/HPI-personal/blob/master/scripts/listens) which just passes an generator (iterator) as the first argument to `query_range` Those can be imported and accept any sort of iterator, `hpi query` just defaults to the output of functions here. As an example, see [`listens`](https://github.com/purarue/HPI-personal/blob/master/scripts/listens) which just passes an generator (iterator) as the first argument to `query_range`

2
doc/SETUP.org
View file

@ -387,7 +387,7 @@ But there is an extra caveat: rexport is already coming with nice [[https://gith
Several other HPI modules are following a similar pattern: hypothesis, instapaper, pinboard, kobo, etc. Several other HPI modules are following a similar pattern: hypothesis, instapaper, pinboard, kobo, etc.
Since the [[https://github.com/karlicoss/rexport#api-limitations][reddit API has limited results]], you can use [[https://github.com/seanbreckenridge/pushshift_comment_export][my.reddit.pushshift]] to access older reddit comments, which both then get merged into =my.reddit.all.comments= Since the [[https://github.com/karlicoss/rexport#api-limitations][reddit API has limited results]], you can use [[https://github.com/purarue/pushshift_comment_export][my.reddit.pushshift]] to access older reddit comments, which both then get merged into =my.reddit.all.comments=
** Twitter ** Twitter

2
misc/.flake8-karlicoss
View file

@ -32,6 +32,6 @@ ignore =
# #
# as a reference: # as a reference:
# https://github.com/seanbreckenridge/cookiecutter-template/blob/master/%7B%7Bcookiecutter.module_name%7D%7D/setup.cfg # https://github.com/purarue/cookiecutter-template/blob/master/%7B%7Bcookiecutter.module_name%7D%7D/setup.cfg
# and this https://github.com/karlicoss/HPI/pull/151 # and this https://github.com/karlicoss/HPI/pull/151
# find ./my | entr flake8 --ignore=E402,E501,E741,W503,E266,E302,E305,E203,E261,E252,E251,E221,W291,E225,E303,E702,E202,F841,E731,E306,E127 E722,E231 my | grep -v __NOT_HPI_MODULE__ # find ./my | entr flake8 --ignore=E402,E501,E741,W503,E266,E302,E305,E203,E261,E252,E251,E221,W291,E225,E303,E702,E202,F841,E731,E306,E127 E722,E231 my | grep -v __NOT_HPI_MODULE__

2
my/browser/active_browser.py
View file

@ -1,5 +1,5 @@
""" """
Parses active browser history by backing it up with [[http://github.com/seanbreckenridge/sqlite_backup][sqlite_backup]] Parses active browser history by backing it up with [[http://github.com/purarue/sqlite_backup][sqlite_backup]]
""" """
REQUIRES = ["browserexport", "sqlite_backup"] REQUIRES = ["browserexport", "sqlite_backup"]

2
my/browser/export.py
View file

@ -1,5 +1,5 @@
""" """
Parses browser history using [[http://github.com/seanbreckenridge/browserexport][browserexport]] Parses browser history using [[http://github.com/purarue/browserexport][browserexport]]
""" """
REQUIRES = ["browserexport"] REQUIRES = ["browserexport"]

10
my/google/takeout/parser.py
View file

@ -1,7 +1,7 @@
""" """
Parses Google Takeout using [[https://github.com/seanbreckenridge/google_takeout_parser][google_takeout_parser]] Parses Google Takeout using [[https://github.com/purarue/google_takeout_parser][google_takeout_parser]]
See [[https://github.com/seanbreckenridge/google_takeout_parser][google_takeout_parser]] for more information See [[https://github.com/purarue/google_takeout_parser][google_takeout_parser]] for more information
about how to export and organize your takeouts about how to export and organize your takeouts
If the DISABLE_TAKEOUT_CACHE environment variable is set, this won't cache individual If the DISABLE_TAKEOUT_CACHE environment variable is set, this won't cache individual
@ -12,7 +12,7 @@ zip files of the exports, which are temporarily unpacked while creating
the cachew cache the cachew cache
""" """
REQUIRES = ["git+https://github.com/seanbreckenridge/google_takeout_parser"] REQUIRES = ["git+https://github.com/purarue/google_takeout_parser"]
import os import os
from collections.abc import Sequence from collections.abc import Sequence
@ -36,7 +36,7 @@ from google_takeout_parser.merge import CacheResults, GoogleEventSet
from google_takeout_parser.models import BaseEvent from google_takeout_parser.models import BaseEvent
from google_takeout_parser.path_dispatch import TakeoutParser from google_takeout_parser.path_dispatch import TakeoutParser
# see https://github.com/seanbreckenridge/dotfiles/blob/master/.config/my/my/config/__init__.py for an example # see https://github.com/purarue/dotfiles/blob/master/.config/my/my/config/__init__.py for an example
from my.config import google as user_config from my.config import google as user_config
@ -123,7 +123,7 @@ def events(disable_takeout_cache: bool = DISABLE_TAKEOUT_CACHE) -> CacheResults:
else: else:
results = exit_stack.enter_context(match_structure(path, expected=EXPECTED, partial=True)) results = exit_stack.enter_context(match_structure(path, expected=EXPECTED, partial=True))
for m in results: for m in results:
# e.g. /home/sean/data/google_takeout/Takeout-1634932457.zip") -> 'Takeout-1634932457' # e.g. /home/username/data/google_takeout/Takeout-1634932457.zip") -> 'Takeout-1634932457'
# means that zipped takeouts have nice filenames from cachew # means that zipped takeouts have nice filenames from cachew
cw_id, _, _ = path.name.rpartition(".") cw_id, _, _ = path.name.rpartition(".")
# each takeout result is cached as well, in individual databases per-type # each takeout result is cached as well, in individual databases per-type

4
my/ip/all.py
View file

@ -3,10 +3,10 @@ An example all.py stub module that provides ip data
To use this, you'd add IP providers that yield IPs to the 'ips' function To use this, you'd add IP providers that yield IPs to the 'ips' function
For an example of how this could be used, see https://github.com/seanbreckenridge/HPI/tree/master/my/ip For an example of how this could be used, see https://github.com/purarue/HPI/tree/master/my/ip
""" """
REQUIRES = ["git+https://github.com/seanbreckenridge/ipgeocache"] REQUIRES = ["git+https://github.com/purarue/ipgeocache"]
from collections.abc import Iterator from collections.abc import Iterator

2
my/ip/common.py
View file

@ -1,5 +1,5 @@
""" """
Provides location/timezone data from IP addresses, using [[https://github.com/seanbreckenridge/ipgeocache][ipgeocache]] Provides location/timezone data from IP addresses, using [[https://github.com/purarue/ipgeocache][ipgeocache]]
""" """
from my.core import __NOT_HPI_MODULE__ # isort: skip from my.core import __NOT_HPI_MODULE__ # isort: skip

2
my/location/fallback/via_ip.py
View file

@ -2,7 +2,7 @@
Converts IP addresses provided by my.location.ip to estimated locations Converts IP addresses provided by my.location.ip to estimated locations
""" """
REQUIRES = ["git+https://github.com/seanbreckenridge/ipgeocache"] REQUIRES = ["git+https://github.com/purarue/ipgeocache"]
from dataclasses import dataclass from dataclasses import dataclass
from datetime import timedelta from datetime import timedelta

2
my/location/google_takeout.py
View file

@ -2,7 +2,7 @@
Extracts locations using google_takeout_parser -- no shared code with the deprecated my.location.google Extracts locations using google_takeout_parser -- no shared code with the deprecated my.location.google
""" """
REQUIRES = ["git+https://github.com/seanbreckenridge/google_takeout_parser"] REQUIRES = ["git+https://github.com/purarue/google_takeout_parser"]
from collections.abc import Iterator from collections.abc import Iterator

2
my/location/google_takeout_semantic.py
View file

@ -5,7 +5,7 @@ Extracts semantic location history using google_takeout_parser
# This is a separate module to prevent ImportError and a new config block from breaking # This is a separate module to prevent ImportError and a new config block from breaking
# previously functional my.location.google_takeout locations # previously functional my.location.google_takeout locations
REQUIRES = ["git+https://github.com/seanbreckenridge/google_takeout_parser"] REQUIRES = ["git+https://github.com/purarue/google_takeout_parser"]
from collections.abc import Iterator from collections.abc import Iterator
from dataclasses import dataclass from dataclasses import dataclass

2
my/location/via_ip.py
View file

@ -1,4 +1,4 @@
REQUIRES = ["git+https://github.com/seanbreckenridge/ipgeocache"] REQUIRES = ["git+https://github.com/purarue/ipgeocache"]
from my.core.warnings import high from my.core.warnings import high

6
my/reddit/pushshift.py
View file

@ -1,11 +1,11 @@
""" """
Gives you access to older comments possibly not accessible with rexport Gives you access to older comments possibly not accessible with rexport
using pushshift using pushshift
See https://github.com/seanbreckenridge/pushshift_comment_export See https://github.com/purarue/pushshift_comment_export
""" """
REQUIRES = [ REQUIRES = [
"git+https://github.com/seanbreckenridge/pushshift_comment_export", "git+https://github.com/purarue/pushshift_comment_export",
] ]
from dataclasses import dataclass from dataclasses import dataclass
@ -21,7 +21,7 @@ from my.core.cfg import make_config
@dataclass @dataclass
class pushshift_config(uconfig.pushshift): class pushshift_config(uconfig.pushshift):
''' '''
Uses [[https://github.com/seanbreckenridge/pushshift_comment_export][pushshift]] to get access to old comments Uses [[https://github.com/purarue/pushshift_comment_export][pushshift]] to get access to old comments
''' '''
# path[s]/glob to the exported JSON data # path[s]/glob to the exported JSON data