Add test demonstrating unloading the module during dynamic configuration
This commit is contained in:
parent
0ac78143f2
commit
1d0ef82d32
3 changed files with 63 additions and 6 deletions
|
@ -20,11 +20,11 @@ Now, the requirements as I see it:
|
||||||
|
|
||||||
1. configuration should be *extremely* flexible
|
1. configuration should be *extremely* flexible
|
||||||
|
|
||||||
We need to make sure it's very easy to combine/filter/extend data without having to modify and rewrite the module code.
|
We need to make sure it's very easy to combine/filter/extend data without having to turn the module code inside out.
|
||||||
This means using a powerful language for config, and realistically, a Turing complete.
|
This means using a powerful language for config, and realistically, a Turing complete.
|
||||||
|
|
||||||
General: that means that you should be able to use powerful syntax, potentially running arbitrary code if
|
General: that means that you should be able to use powerful syntax, potentially running arbitrary code if
|
||||||
this is something you need (for whatever mad reason). It should be possible to override config attributes in runtime, if necessary.
|
this is something you need (for whatever mad reason). It should be possible to override config attributes *in runtime*, if necessary, without rewriting files on the filesystem.
|
||||||
|
|
||||||
Specific: we've got Python already, so it makes a lot of sense to use it!
|
Specific: we've got Python already, so it makes a lot of sense to use it!
|
||||||
|
|
||||||
|
@ -160,6 +160,16 @@ from my.config import bluemaestro as user_config
|
||||||
Let's go through requirements:
|
Let's go through requirements:
|
||||||
|
|
||||||
- (1): *yes*, simply importing Python code is the most flexible you can get
|
- (1): *yes*, simply importing Python code is the most flexible you can get
|
||||||
|
In addition, in runtime, you can simply assign a new config if you need some dynamic hacking:
|
||||||
|
|
||||||
|
#+begin_src python
|
||||||
|
class new_config:
|
||||||
|
export_path = '/some/hacky/dynamic/path'
|
||||||
|
my.config = new_config
|
||||||
|
#+end_src
|
||||||
|
|
||||||
|
After that, =my.bluemaestro= would run against your new config.
|
||||||
|
|
||||||
- (2): *no*, but backwards compatibility is not necessary in the first version of the module
|
- (2): *no*, but backwards compatibility is not necessary in the first version of the module
|
||||||
- (3): *mostly*, although optional fields require extra work
|
- (3): *mostly*, although optional fields require extra work
|
||||||
- (4): *yes*, whatever is in the config can immediately be used by the code
|
- (4): *yes*, whatever is in the config can immediately be used by the code
|
||||||
|
@ -176,7 +186,7 @@ I see mypy annotations as the only sane way to support it, because we also get (
|
||||||
However, it's using plain files and doesn't satisfy (1).
|
However, it's using plain files and doesn't satisfy (1).
|
||||||
|
|
||||||
Also not sure about (5). =file-config= allows using mypy annotations, but I'm not convinced they would be correctly typed with mypy, I think you need a plugin for that.
|
Also not sure about (5). =file-config= allows using mypy annotations, but I'm not convinced they would be correctly typed with mypy, I think you need a plugin for that.
|
||||||
|
|
||||||
- [[https://mypy.readthedocs.io/en/stable/protocols.html#simple-user-defined-protocols][Protocol]]
|
- [[https://mypy.readthedocs.io/en/stable/protocols.html#simple-user-defined-protocols][Protocol]]
|
||||||
|
|
||||||
I experimented with ~Protocol~ [[https://github.com/karlicoss/HPI/pull/45/commits/90b9d1d9c15abe3944913add5eaa5785cc3bffbc][here]].
|
I experimented with ~Protocol~ [[https://github.com/karlicoss/HPI/pull/45/commits/90b9d1d9c15abe3944913add5eaa5785cc3bffbc][here]].
|
||||||
|
@ -205,7 +215,7 @@ I see mypy annotations as the only sane way to support it, because we also get (
|
||||||
|
|
||||||
Downsides:
|
Downsides:
|
||||||
- we partially lost (5), because dynamic attributes are not transparent to mypy.
|
- we partially lost (5), because dynamic attributes are not transparent to mypy.
|
||||||
|
|
||||||
|
|
||||||
My conclusion was using a *combined approach*:
|
My conclusion was using a *combined approach*:
|
||||||
|
|
||||||
|
@ -214,7 +224,7 @@ My conclusion was using a *combined approach*:
|
||||||
|
|
||||||
Inheritance is a standard mechanism, which doesn't require any extra frameworks and plays well with other Python concepts. As a specific example:
|
Inheritance is a standard mechanism, which doesn't require any extra frameworks and plays well with other Python concepts. As a specific example:
|
||||||
|
|
||||||
#+begin_src python
|
,#+begin_src python
|
||||||
from my.config import bluemaestro as user_config
|
from my.config import bluemaestro as user_config
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
|
@ -264,6 +274,11 @@ Downsides:
|
||||||
|
|
||||||
However, it's extracted in a generic helper, and [[https://github.com/karlicoss/HPI/blob/d6f071e3b12ba1cd5a86ad80e3821bec004e6a6d/my/twitter/archive.py#L17][ends up pretty simple]]
|
However, it's extracted in a generic helper, and [[https://github.com/karlicoss/HPI/blob/d6f071e3b12ba1cd5a86ad80e3821bec004e6a6d/my/twitter/archive.py#L17][ends up pretty simple]]
|
||||||
|
|
||||||
|
- inheriting from ~user_config~ requires it to be a =class= rather than an =object=
|
||||||
|
|
||||||
|
A practical downside is you can't use something like ~SimpleNamespace~.
|
||||||
|
But considering you can define an ad-hoc =class= anywhere, this is fine?
|
||||||
|
|
||||||
My conclusion is that I'm going with this approach for now.
|
My conclusion is that I'm going with this approach for now.
|
||||||
Note that at no stage in required any changes to the user configs, so if I missed something, it would be reversible.
|
Note that at no stage in required any changes to the user configs, so if I missed something, it would be reversible.
|
||||||
|
|
||||||
|
|
|
@ -1,6 +1,8 @@
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
# TODO switch these from using SimpleNamespace
|
||||||
|
|
||||||
def setup_notes_path(notes: Path) -> None:
|
def setup_notes_path(notes: Path) -> None:
|
||||||
# TODO reuse doc from my.cfg?
|
# TODO reuse doc from my.cfg?
|
||||||
from my.cfg import config
|
from my.cfg import config
|
||||||
|
|
|
@ -1,10 +1,11 @@
|
||||||
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from more_itertools import ilen
|
from more_itertools import ilen
|
||||||
|
|
||||||
# TODO NOTE: this wouldn't work because of an early my.config.demo import
|
# TODO NOTE: this wouldn't work because of an early my.config.demo import
|
||||||
# from my.demo import items
|
# from my.demo import items
|
||||||
|
|
||||||
def test_dynamic_config(tmp_path: Path) -> None:
|
def test_dynamic_config_1(tmp_path: Path) -> None:
|
||||||
import my.config
|
import my.config
|
||||||
|
|
||||||
class user_config:
|
class user_config:
|
||||||
|
@ -17,7 +18,46 @@ def test_dynamic_config(tmp_path: Path) -> None:
|
||||||
assert item1.username == 'user'
|
assert item1.username == 'user'
|
||||||
|
|
||||||
|
|
||||||
|
# exactly the same test, but using a different config, to test out the behavious w.r.t. import order
|
||||||
|
def test_dynamic_config_2(tmp_path: Path) -> None:
|
||||||
|
# doesn't work without it!
|
||||||
|
# because the config from test_dybamic_config_1 is cached in my.demo.demo
|
||||||
|
del sys.modules['my.demo']
|
||||||
|
|
||||||
|
import my.config
|
||||||
|
|
||||||
|
class user_config:
|
||||||
|
username = 'user2'
|
||||||
|
data_path = f'{tmp_path}/*.json'
|
||||||
|
my.config.demo = user_config # type: ignore[misc, assignment]
|
||||||
|
|
||||||
|
from my.demo import items
|
||||||
|
[item1, item2] = items()
|
||||||
|
assert item1.username == 'user2'
|
||||||
|
|
||||||
|
|
||||||
import pytest # type: ignore
|
import pytest # type: ignore
|
||||||
|
|
||||||
|
@pytest.mark.skip(reason="won't work at the moment because of inheritance")
|
||||||
|
def test_dynamic_config_simplenamespace(tmp_path: Path) -> None:
|
||||||
|
# doesn't work without it!
|
||||||
|
# because the config from test_dybamic_config_1 is cached in my.demo.demo
|
||||||
|
del sys.modules['my.demo']
|
||||||
|
|
||||||
|
import my.config
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
user_config = SimpleNamespace(
|
||||||
|
username='user3',
|
||||||
|
data_path=f'{tmp_path}/*.json',
|
||||||
|
)
|
||||||
|
my.config.demo = user_config # type: ignore[misc, assignment]
|
||||||
|
|
||||||
|
from my.demo import config
|
||||||
|
assert config().username == 'user3'
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
@pytest.fixture(autouse=True)
|
||||||
def prepare(tmp_path: Path):
|
def prepare(tmp_path: Path):
|
||||||
(tmp_path / 'data.json').write_text('''
|
(tmp_path / 'data.json').write_text('''
|
||||||
|
|
Loading…
Add table
Reference in a new issue