Add test demonstrating unloading the module during dynamic configuration

This commit is contained in:
Dima Gerasimov 2020-05-10 21:57:55 +01:00
parent 0ac78143f2
commit 1d0ef82d32
3 changed files with 63 additions and 6 deletions

View file

@ -20,11 +20,11 @@ Now, the requirements as I see it:
1. configuration should be *extremely* flexible
We need to make sure it's very easy to combine/filter/extend data without having to modify and rewrite the module code.
We need to make sure it's very easy to combine/filter/extend data without having to turn the module code inside out.
This means using a powerful language for config, and realistically, a Turing complete.
General: that means that you should be able to use powerful syntax, potentially running arbitrary code if
this is something you need (for whatever mad reason). It should be possible to override config attributes in runtime, if necessary.
this is something you need (for whatever mad reason). It should be possible to override config attributes *in runtime*, if necessary, without rewriting files on the filesystem.
Specific: we've got Python already, so it makes a lot of sense to use it!
@ -160,6 +160,16 @@ from my.config import bluemaestro as user_config
Let's go through requirements:
- (1): *yes*, simply importing Python code is the most flexible you can get
In addition, in runtime, you can simply assign a new config if you need some dynamic hacking:
#+begin_src python
class new_config:
export_path = '/some/hacky/dynamic/path'
my.config = new_config
#+end_src
After that, =my.bluemaestro= would run against your new config.
- (2): *no*, but backwards compatibility is not necessary in the first version of the module
- (3): *mostly*, although optional fields require extra work
- (4): *yes*, whatever is in the config can immediately be used by the code
@ -176,7 +186,7 @@ I see mypy annotations as the only sane way to support it, because we also get (
However, it's using plain files and doesn't satisfy (1).
Also not sure about (5). =file-config= allows using mypy annotations, but I'm not convinced they would be correctly typed with mypy, I think you need a plugin for that.
- [[https://mypy.readthedocs.io/en/stable/protocols.html#simple-user-defined-protocols][Protocol]]
I experimented with ~Protocol~ [[https://github.com/karlicoss/HPI/pull/45/commits/90b9d1d9c15abe3944913add5eaa5785cc3bffbc][here]].
@ -205,7 +215,7 @@ I see mypy annotations as the only sane way to support it, because we also get (
Downsides:
- we partially lost (5), because dynamic attributes are not transparent to mypy.
My conclusion was using a *combined approach*:
@ -214,7 +224,7 @@ My conclusion was using a *combined approach*:
Inheritance is a standard mechanism, which doesn't require any extra frameworks and plays well with other Python concepts. As a specific example:
#+begin_src python
,#+begin_src python
from my.config import bluemaestro as user_config
@dataclass
@ -264,6 +274,11 @@ Downsides:
However, it's extracted in a generic helper, and [[https://github.com/karlicoss/HPI/blob/d6f071e3b12ba1cd5a86ad80e3821bec004e6a6d/my/twitter/archive.py#L17][ends up pretty simple]]
- inheriting from ~user_config~ requires it to be a =class= rather than an =object=
A practical downside is you can't use something like ~SimpleNamespace~.
But considering you can define an ad-hoc =class= anywhere, this is fine?
My conclusion is that I'm going with this approach for now.
Note that at no stage in required any changes to the user configs, so if I missed something, it would be reversible.

View file

@ -1,6 +1,8 @@
from pathlib import Path
# TODO switch these from using SimpleNamespace
def setup_notes_path(notes: Path) -> None:
# TODO reuse doc from my.cfg?
from my.cfg import config

View file

@ -1,10 +1,11 @@
import sys
from pathlib import Path
from more_itertools import ilen
# TODO NOTE: this wouldn't work because of an early my.config.demo import
# from my.demo import items
def test_dynamic_config(tmp_path: Path) -> None:
def test_dynamic_config_1(tmp_path: Path) -> None:
import my.config
class user_config:
@ -17,7 +18,46 @@ def test_dynamic_config(tmp_path: Path) -> None:
assert item1.username == 'user'
# exactly the same test, but using a different config, to test out the behavious w.r.t. import order
def test_dynamic_config_2(tmp_path: Path) -> None:
# doesn't work without it!
# because the config from test_dybamic_config_1 is cached in my.demo.demo
del sys.modules['my.demo']
import my.config
class user_config:
username = 'user2'
data_path = f'{tmp_path}/*.json'
my.config.demo = user_config # type: ignore[misc, assignment]
from my.demo import items
[item1, item2] = items()
assert item1.username == 'user2'
import pytest # type: ignore
@pytest.mark.skip(reason="won't work at the moment because of inheritance")
def test_dynamic_config_simplenamespace(tmp_path: Path) -> None:
# doesn't work without it!
# because the config from test_dybamic_config_1 is cached in my.demo.demo
del sys.modules['my.demo']
import my.config
from types import SimpleNamespace
user_config = SimpleNamespace(
username='user3',
data_path=f'{tmp_path}/*.json',
)
my.config.demo = user_config # type: ignore[misc, assignment]
from my.demo import config
assert config().username == 'user3'
@pytest.fixture(autouse=True)
def prepare(tmp_path: Path):
(tmp_path / 'data.json').write_text('''