153 lines
6.6 KiB
Org Mode
153 lines
6.6 KiB
Org Mode
I feel like it's good to keep the rationales in the documentation,
|
|
but happy to [[https://github.com/karlicoss/HPI/issues/46][discuss]] it here.
|
|
|
|
Before discussing the abstract matters, let's consider a specific situation.
|
|
Say, we want to let the user configure [[https://github.com/karlicoss/HPI/blob/master/my/bluemaestro/__init__.py][bluemaestro]] module.
|
|
At the moment, it uses the following config attributes:
|
|
|
|
- ~export_path~
|
|
|
|
Path to the data, this is obviously a *required* attribute
|
|
|
|
- ~cache_path~
|
|
|
|
Cache is extremely useful to speed up some queries. But it's *optional*, everything should work without it.
|
|
|
|
|
|
|
|
I'll refer to this config as *specific* further in the doc, and give examples. to each point. Note that they are only illustrating the specific requirement, potentially ignoring the other ones.
|
|
Now, the requirements as I see it:
|
|
|
|
1. configuration should be *extremely* flexible
|
|
|
|
We need to make sure it's very easy to combine/filter/extend data without having to modify and rewrite the module code.
|
|
This means using a powerful language for config, and realistically, a Turing complete.
|
|
|
|
General: that means that you should be able to use powerful, potentially running arbitrary code if
|
|
this is something
|
|
|
|
Specific: we've got Python already, so it makes a lot of sense to use it!
|
|
|
|
#+begin_src python
|
|
class bluemaestro:
|
|
export_path = '/path/to/bluemaestro'
|
|
cache_path = '/tmp/bluemaestro.cache'
|
|
#+end_src
|
|
|
|
Downsides:
|
|
|
|
- keeping it Turing complete means it's potentially less accessible to people less familiar with programming
|
|
|
|
But see the further point about keeping it simple. I claim that simple programs look as easy as simple json.
|
|
|
|
- Python is 'less safe' than a plain json/yaml config
|
|
|
|
But at the moment the whole thing is running potentially untrusted Python code anyway.
|
|
It's not a tool you're going to install it across your organization, run under root privileges, and let the employers tweak it.
|
|
|
|
Ultimately, you set it up for yourself, and the config has exactly the same permissions as the code you're installing.
|
|
Thinking that plain config would give you more security is deceptive, and it's a false sense of security (at this stage of the project).
|
|
|
|
# TODO I don't mind having json/toml/whatever, but only as an additional interface
|
|
|
|
I also write more about all this [[https://beepb00p.xyz/configs-suck.html][here]].
|
|
|
|
2. configuration should be *backwards compatible*
|
|
|
|
General: the whole system is pretty chaotic, it's hard to control the versioning of different modules and their compatibility.
|
|
It's important to allow changing attribute names and adding new functionality, while making sure the module works against an older version of the config.
|
|
Ideally warn the user that they'd better migrate to a newer version if the fallbacks are triggered.
|
|
Potentially: use individual versions for modules? Although it makes things a bit complicated.
|
|
|
|
Specific: say the module is using a new config attribute, ~timezone~.
|
|
We would need to adapt the module to support the old configs without timezone. For example, in ~bluemaestro.py~ (pseudocode):
|
|
|
|
#+begin_src python
|
|
user_config = load_user_config()
|
|
if not hasattr(user_config, 'timezone'):
|
|
warnings.warn("Please specify 'timezone' in the config! Falling back to the system timezone.")
|
|
user_config.timezonee = get_system_timezone()
|
|
#+end_src
|
|
|
|
This is possible to achieve with pretty much any config format, just important to keep in mind.
|
|
|
|
3. configuration should be as *easy to write* as possible
|
|
|
|
General: as lean and non-verbose as possible. No extra imports, no extra inheritance, annotations, etc.
|
|
|
|
Specific: the user *only* has to specify ~export_path~ to make the module function and that's it. For example:
|
|
|
|
#+begin_src js
|
|
{
|
|
'export_path': '/path/to/bluemaestro/'
|
|
}
|
|
#+end_src
|
|
|
|
It's possible to achieve with any configuration format (aided by some helpers to fill in optional attributes etc), so it's more of a guiding principle.
|
|
|
|
4. configuration should be as *easy to use and extend* as possible
|
|
|
|
General: enable the users to add new config attributes and *immediately* use them without any hassle and boilerplate.
|
|
It's easy to achieve on it's own, but harder to achieve simultaneously with (2).
|
|
|
|
Specific: if you keep the config as Python, simply importing the config in the module satisfies this property:
|
|
|
|
#+begin_src python
|
|
from my.config import bluemaestro as user_config
|
|
#+end_src
|
|
|
|
If the config is in JSON or something, it's possible to load it dynamically too without the boilerplate.
|
|
|
|
5. configuration should have checks
|
|
|
|
General: make sure it's easy to track down configuration errors. At least runtime checks for required attributes, their types, warnings, that sort of thing. But a biggie for me is using *mypy* to statically typecheck the modules.
|
|
To some extent it gets in the way of (2) and (4).
|
|
|
|
Specific: using ~NamedTuple/dataclass~ has capabilities to verify the config with no extra boilerplate on the user side.
|
|
|
|
#+begin_src python
|
|
class bluemaestro(NamedTuple):
|
|
export_path: str
|
|
cache_path : Optional[str] = None
|
|
|
|
raw_config = json.load('configs/bluemaestro.json')
|
|
config = bluemaestro(**raw_config)
|
|
#+end_src
|
|
|
|
This will fail if required =export_path= is missing, and fill optional =cache_path= with None. In addition, it's ~mypy~ friendly.
|
|
|
|
6. configuration should be easy to document
|
|
|
|
General: ideally, it should be autogenerated, be self-descriptive and have some sort of schema, to make sure the documentation (which no one likes to write) doesn't diverge.
|
|
|
|
Specific: mypy annotations seem like the way to go. I did some experiments with using [[https://github.com/karlicoss/HPI/pull/45/commits/90b9d1d9c15abe3944913add5eaa5785cc3bffbc][Protocol]] or a [[https://github.com/karlicoss/HPI/pull/45/commits/c877104b90c9d168eaec96e0e770e59048ce4465][NamedTuple]] for a self-descriptive ~my.reddit~ configuration.
|
|
See the example from (5), it's pretty clear from the code what needs to be in the config.
|
|
|
|
|
|
* Solutions?
|
|
|
|
# different stages
|
|
# TODO keep it chaotic
|
|
# make it safer
|
|
# TODO add defensiveness
|
|
|
|
- file-config https://github.com/karlicoss/HPI/issues/12#issuecomment-610038961
|
|
no mypy?
|
|
|
|
|
|
|
|
* Side modules :noexport:
|
|
|
|
Some of TODO rexport?
|
|
|
|
To some extent, this is an experiment. I'm not sure how much value is in .
|
|
|
|
|
|
One thing are TODO software? libraries that have fairly well defined APIs and you can reasonably version them.
|
|
|
|
Another thing is the modules for accessing data, where you'd hopefully have everything backwards compatible.
|
|
Maybe in the future
|
|
|
|
I'm just not sure, happy to hear people's opinions on this.
|
|
|
|
|