HPI/doc/CONFIGURING.org

I feel like it's good to keep the rationales in the documentation,
but happy to [[https://github.com/karlicoss/HPI/issues/46][discuss]] it here.

Before discussing the abstract matters, let's consider a specific situation.
Say, we want to let the user configure [[https://github.com/karlicoss/HPI/blob/master/my/bluemaestro/__init__.py][bluemaestro]] module.
At the moment, it uses the following config attributes:

- ~export_path~

  Path to the data, this is obviously a *required* attribute

- ~cache_path~

  Cache is extremely useful to speed up some queries. But it's *optional*, everything should work without it.



I'll refer to this config as *specific* further in the doc, and give examples. to each point. Note that they are only illustrating the specific requirement, potentially ignoring the other ones.
Now, the requirements as I see it:

1. configuration should be *extremely* flexible

   We need to make sure it's very easy to combine/filter/extend data without having to modify and rewrite the module code.
   This means using a powerful language for config, and realistically, a Turing complete.

   General: that means that you should be able to use powerful, potentially running arbitrary code if
   this is something

   Specific: we've got Python already, so it makes a lot of sense to use it!

   #+begin_src python
   class bluemaestro:
       export_path = '/path/to/bluemaestro/data'
       cache_path  = '/tmp/bluemaestro.cache'
   #+end_src

   Downsides:

   - keeping it overly flexible and powerful means it's potentially less accessible to people less familiar with programming

     But see the further point about keeping it simple. I claim that simple programs look as easy as simple json.

   - Python is 'less safe' than a plain json/yaml config

     But at the moment the whole thing is running potentially untrusted Python code anyway.
     It's not a tool you're going to install it across your organization, run under root privileges, and let the employers tweak it.

     Ultimately, you set it up for yourself, and the config has exactly the same permissions as the code you're installing.
     Thinking that plain config would give you more security is deceptive, and it's a false sense of security (at this stage of the project).

   # TODO  I don't mind having  json/toml/whatever, but only as an additional interface

   I also write more about all this [[https://beepb00p.xyz/configs-suck.html][here]].

2. configuration should be *backwards compatible*

   General: the whole system is pretty chaotic, it's hard to control the versioning of different modules and their compatibility.
   It's important to allow changing attribute names and adding new functionality, while making sure the module works against an older version of the config.
   Ideally warn the user that they'd better migrate to a newer version if the fallbacks are triggered.
   Potentially: use individual versions for modules? Although it makes things a bit complicated.

   Specific: say the module is using a new config attribute, ~timezone~.
   We would need to adapt the module to support the old configs without timezone. For example, in ~bluemaestro.py~ (pseudocode):

   #+begin_src python
   user_config = load_user_config()
   if not hasattr(user_config, 'timezone'):
       warnings.warn("Please specify 'timezone' in the config! Falling back to the system timezone.")
       user_config.timezonee = get_system_timezone()
   #+end_src

   This is possible to achieve with pretty much any config format, just important to keep in mind.

   Downsides: hopefully no one argues backwards compatibility is important.

3. configuration should be as *easy to write* as possible

   General: as lean and non-verbose as possible. No extra imports, no extra inheritance, annotations, etc. Loose coupling.

   Specific: the user *only* has to specify ~export_path~ to make the module function and that's it. For example:

   #+begin_src js
   {
        'export_path': '/path/to/bluemaestro/'
   }
   #+end_src

   It's possible to achieve with any configuration format (aided by some helpers to fill in optional attributes etc), so it's more of a guiding principle.

   Downsides:

   - no (mandatory) annotations means more potential to break, but I'd rather leave this decision to the users

4. configuration should be as *easy to use and extend* as possible

   General: enable the users to add new config attributes and *immediately* use them without any hassle and boilerplate.
   It's easy to achieve on it's own, but harder to achieve simultaneously with (2).

   Specific: if you keep the config as Python, simply importing the config in the module satisfies this property:

   #+begin_src python
   from my.config import bluemaestro as user_config
   #+end_src

   If the config is in JSON or something, it's possible to load it dynamically too without the boilerplate.

   Downsides: none, hopefully no one is against extensibility

5. configuration should have checks

   General: make sure it's easy to track down configuration errors. At least runtime checks for required attributes, their types, warnings, that sort of thing. But a biggie for me is using *mypy* to statically typecheck the modules.
   To some extent it gets in the way of (2) and (4).

   Specific: using ~NamedTuple/dataclass~ has capabilities to verify the config with no extra boilerplate on the user side.

   #+begin_src python
   class bluemaestro(NamedTuple):
        export_path: str
        cache_path : Optional[str] = None

   raw_config = json.load('configs/bluemaestro.json')
   config = bluemaestro(**raw_config)
   #+end_src

   This will fail if required =export_path= is missing, and fill optional =cache_path= with None. In addition, it's ~mypy~ friendly.

   Downsides: none, especially if it's possbile to turn checks on/off.

6. configuration should be easy to document

   General: ideally, it should be autogenerated, be self-descriptive and have some sort of schema, to make sure the documentation (which no one likes to write) doesn't diverge.

   Specific: mypy annotations seem like the way to go. See the example from (5), it's pretty clear from the code what needs to be in the config.

   Downsides: none, self-documented code is good.

* Solution?

Now I'll consider potential solutions to the configuration, taking the different requirements into account.

Like I already mentiond, plain configs (JSON/YAML/TOML) are very inflexible and go against (1), which in my opinion think makes them no-go.

So: my suggestion is to write the *configs as Python code*.
It's hard to satisfy all requirements *at the same time*, but I want to argue, it's possible to satisfy most of them, depending on the maturity of the module which we're configuring.

Let's say you want to write a new module. You start with a

#+begin_src python
class bluemaestro:
    export_path = '/path/to/bluemaestro/data'
    cache_path  = '/tmp/bluemaestro.cache'
#+end_src

And to use it:

#+begin_src python
from my.config import bluemaestro as user_config
#+end_src

Let's go through requirements:

- (1): *yes*, simply importing Python code is the most flexible you can get
- (2): *no*, but backwards compatibility is not necessary in the first version of the module
- (3): *mostly*, although optional fields require extra work
- (4): *yes*, whatever is in the config can immediately be used by the code
- (5): *mostly*, imports are transparent to ~mypy~, although runtime type checks would be nice too
- (6): *no*, you have to guess the config from the usage.

This approach is extremely simple, and already *good enough for initial prototyping* or *private modules*.

The main downside so far is the lack of documentation (6), which I'll try to solve next.
I see mypy annotations as the only sane way to support it, so we could use:

- potentially [[https://github.com/karlicoss/HPI/issues/12#issuecomment-610038961][file-config]]

  However, it's using plain files and doesn't satisfy (1).

  Also not sure about (5). =file-config= allows using mypy annotations, but I'm not convinced they would be correctly typed with mypy, I think you need a plugin for that.
 
- [[https://mypy.readthedocs.io/en/stable/protocols.html#simple-user-defined-protocols][Protocol]]

  I experimented with ~Protocol~ [[https://github.com/karlicoss/HPI/pull/45/commits/90b9d1d9c15abe3944913add5eaa5785cc3bffbc][here]].
  It's pretty cool, very flexible, and doesn't impose any runtime modifications, which makes it good for (4).

  The downsides are:

  - it doesn't support optional attributes (optional as in non-required, not as ~typing.Optional~), so it goes against (3)
  - prior to python 3.8, it's a part of =typing_extensions= rather than standard =typing=, so using it requires guarding the code with =if typing.TYPE_CHECKING=, which is a bit confusing and bloating.

- =NamedTuple=

  [[https://github.com/karlicoss/HPI/pull/45/commits/c877104b90c9d168eaec96e0e770e59048ce4465][Here]] I experimented with using ~NamedTuple~.

  Similarly to Protocol, it's self-descriptive, and in addition allows for non-required fields.
  # TODO something about helper methods? can't use them with Protocol

  Downsides:
  - it goes against (4), because NamedTuple can only contain the attributes declared in the schema.

My conclusion was using a combined approach.



* Side modules :noexport:

Some of TODO rexport?

To some extent, this is an experiment. I'm not sure how much value is in .


One thing are TODO software? libraries that have fairly well defined APIs and you can reasonably version them.

Another thing is the modules for accessing data, where you'd hopefully have everything backwards compatible.
Maybe in the future

I'm just not sure, happy to hear people's opinions on this.