HPI/doc/CONFIGURING.org

• Side modules

I feel like it's good to keep the rationales in the documentation, but happy to discuss it here.

Before discussing the abstract matters, let's consider a specific situation. Say, we want to let the user configure bluemaestro module. At the moment, it uses the following config attributes:

• export_path Path to the data, this is obviously a required attribute
• cache_path Cache is extremely useful to speed up some queries. But it's optional, everything should work without it.

I'll refer to this config as specific further in the doc.

Now, the requirements I see, approximately in the order of decreasing importance (at least as I see it):

1. configuration should be extremely flexible

   We need to make sure it's very easy to combine/filter/extend data without having to modify and rewrite the module code. This means using a powerful language for config, and realistically, a Turing complete.

   General: that means that you should be able to use powerful, potentially running arbitrary code if this is something

   Specific: we've got Python already, so it makes a lot of sense to use it!

   class bluemaestro:
       export_path = '/path/to/bluemaestro'
       cache_path  = '/tmp/bluemaestro.cache'

   Downsides:

   • keeping it Turing complete means it's potentially less accessible to people less familiar with programming But see the next point about keeping it simple. I claim that simple programs look as easy as simple json.
   • Python is 'less safe' than a plain json/yaml config But at the moment the whole thing is running potentially untrusted Python code anyway. It's not a tool you're going to install it across your organization, run under root privileges, and let the employers tweak it. Ultimately, you set it up for yourself, and the config has exactly the same permissions as the code you're installing. Thinking that plain config would give you more security is deceptive, and it's a false sense of security (at this stage of the project).

   I also write more about all this here.

2. configuration should be as easy as possible

   General: as lean and non-verbose as possible. No extra imports, no extra inheritance, annotations, etc.

   Specific: the user only has to specify export_path to make the module function and that's it. For example:

   {
        'export_path': '/path/to/bluemaestro/'
   }

   JSON (aided by some helpers to fill in optional attributes etc) satisfies this property

• usable with mypy, TODO imports work I'm very opinionated about this.
• as little dynamic stuff as possible

90b9d1d9c1

file-config https://github.com/karlicoss/HPI/issues/12#issuecomment-610038961

no mypy?

Side modules

Some of TODO rexport?

To some extent, this is an experiment. I'm not sure how much value is in .

One thing are TODO software? libraries that have fairly well defined APIs and you can reasonably version them.

Another thing is the modules for accessing data, where you'd hopefully have everything backwards compatible. Maybe in the future

I'm just not sure, happy to hear people's opinions on this.