fz0x1/HPI
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

denylist docs/installation instructions

Browse source
This commit is contained in:
Sean Breckenridge 2023-02-21 16:27:55 -08:00
parent 8971a3915e
commit 6d4724ffbc
3 changed files with 130 additions and 100 deletions
Download patch file Download diff file Expand all files Collapse all files

99
my/core/denylist.py
View file

@ -1,105 +1,8 @@
"""
TODO: move this to doc/DENYLIST ?
A helper module for defining denylists for sources programatically
(in lamens terms, this lets you remove some output from a module you don't want)
Lets you specify a class, an attribute to match on,
and a json file containing a list of values to deny/filter out
As an example, for a class like this:
class IP(NamedTuple):
ip: str
dt: datetime
A possible denylist file would contain:
[
{
"ip": "192.168.1.1",
},
{
"dt": "2020-06-02T03:12:00+00:00",
}
]
Note that if the value being compared to is not a single (non-array/object) JSON primitive
(str, int, float, bool, None), it will be converted to a string before comparison
To use this in code:
```
from my.ip.all import ips
filtered = DenyList("~/data/ip_denylist.json").filter(ips())
```
To add items to the denylist, in python (in a one-off script):
```
from my.ip.all import ips
from my.core.denylist import DenyList
d = DenyList("~/data/ip_denylist.json")
for ip in ips():
# some custom code you define
if ip.ip == ...:
d.deny(key="ip", value=ip.ip)
d.write()
```
... or interactively, which requires `fzf` to be installed, after running
```
from my.ip.all import ips
from my.core.denylist import DenyList
d = DenyList("~/data/ip_denylist.json")
d.deny_cli(ips())
d.write()
```
This is meant for relatively simple filters, where you want to filter out
based on a single attribute of a namedtuple/dataclass. If you want to do something
more complex, I would recommend overriding the all.py file for that source and
writing your own filter function there.
For more info on all.py:
https://github.com/karlicoss/HPI/blob/master/doc/MODULE_DESIGN.org#allpy
This would typically be used in an overriden all.py file, or in a one-off script
which you may want to filter out some items from a source, progressively adding more
items to the denylist as you go.
A potential my/ip/all.py file might look like:
```
from typing import Iterator
from my.ip.common import IP # type: ignore[import]
from my.core.denylist import DenyList
deny = DenyList("~/data/ip_denylist.json")
def ips() -> Iterator[IP]:
from my.ip import discord
yield from deny.filter(discord.ips())
```
To add items to the denylist, you could create a __main__.py file, or:
```
python3 -c 'from my.ip import all; all.deny.deny_cli(all.ips())'
```
Sidenote: the reason why we want to specifically override
the all.py and not just create a script that filters out the items you're
not interested in is because we want to be able to import from `my.ip.all`
or `my.location.all` from other modules and get the filtered results, without
having to mix data filtering logic with parsing/loading/caching (the stuff HPI does)
For docs, see doc/DENYLIST.md
"""
import sys