Rename my_configuration to mycfg for brevity

This commit is contained in:
Dima Gerasimov 2019-12-19 19:45:25 +00:00 committed by Dmitrii Gerasimov
parent cd804091c3
commit 066641a4ce
31 changed files with 65 additions and 70 deletions

View file

@ -15,7 +15,7 @@ jobs:
# - run: python3 -m mypy --namespace-packages my # - run: python3 -m mypy --namespace-packages my
# - run: python3 -m pylint -E my # - run: python3 -m pylint -E my
# TODO need to keep full my_configuration.py as example? to dummy run CI # TODO need to keep full mycfg.py as example? to dummy run CI

View file

@ -25,17 +25,17 @@ Short example to give you an idea: "which subreddits I find most interesting?"
| QuantifiedSelf | 28 | | QuantifiedSelf | 28 |
* Setting up * Setting up
** =my_configuration= package for private paths/repositories (optional) ** =mycfg= package for private paths/repositories (optional)
If you're not planning to use private configuration (some modules don't need it) you can skip straight to the next step. Still, I'd recommend you to read anyway. If you're not planning to use private configuration (some modules don't need it) you can skip straight to the next step. Still, I'd recommend you to read anyway.
First you need to tell the package where to look for your data and external repositories, which is done though a separate (private) package named ~my_configuration~. First you need to tell the package where to look for your data and external repositories, which is done though a separate (private) package named ~mycfg~.
You can see example in ~my_configuration_template~. You can copy it somewhere else and modify to your needs. You can see example in ~mycfg_template~. You can copy it somewhere else and modify to your needs.
Some explanations: Some explanations:
#+begin_src bash :exports results :results output #+begin_src bash :exports results :results output
for x in $(find my_configuration_template/ | grep -v -E 'mypy_cache|.git|__pycache__'); do for x in $(find mycfg_template/ | grep -v -E 'mypy_cache|.git|__pycache__'); do
if [[ -L "$x" ]]; then if [[ -L "$x" ]]; then
echo "l $x -> $(readlink $x)" echo "l $x -> $(readlink $x)"
elif [[ -d "$x" ]]; then elif [[ -d "$x" ]]; then
@ -49,9 +49,9 @@ Some explanations:
#+RESULTS: #+RESULTS:
#+begin_example #+begin_example
d my_configuration_template/ d mycfg_template/
d my_configuration_template/my_configuration d mycfg_template/mycfg
f my_configuration_template/my_configuration/__init__.py f mycfg_template/mycfg/__init__.py
--- ---
class paths: class paths:
""" """
@ -60,15 +60,15 @@ f my_configuration_template/my_configuration/__init__.py
class hypexport: class hypexport:
export_dir = '/tmp/my_demo/backups/hypothesis' export_dir = '/tmp/my_demo/backups/hypothesis'
--- ---
d my_configuration_template/my_configuration/repos d mycfg_template/mycfg/repos
l my_configuration_template/my_configuration/repos/hypexport -> /tmp/my_demo/hypothesis_repo l mycfg_template/mycfg/repos/hypexport -> /tmp/my_demo/hypothesis_repo
#+end_example #+end_example
As you can see, generally you specify fixed paths (e.g. to backup directory) in ~__init__.py~. As you can see, generally you specify fixed paths (e.g. to backup directory) in ~__init__.py~.
Feel free to add other files as well though to organize better, it's a real python package after all! Feel free to add other files as well though to organize better, it's a real python package after all!
Some things (e.g. links to external packages like [[https://github.com/karlicoss/hypexport][hypexport]]) are specified as normal symlinks in ~repos~ directory. Some things (e.g. links to external packages like [[https://github.com/karlicoss/hypexport][hypexport]]) are specified as normal symlinks in ~repos~ directory.
That way you get easy imports (e.g. =import my_configuration.repos.hypexport.model=) and proper IDE integration. That way you get easy imports (e.g. =import mycfg.repos.hypexport.model=) and proper IDE integration.
# TODO link to post about exports? # TODO link to post about exports?
** =with_my= helper script ** =with_my= helper script
@ -76,7 +76,7 @@ Next, point =with_my= script to your private configuration:
#+begin_src bash #+begin_src bash
cp with_my.example with_my cp with_my.example with_my
vim with_my # specify path to your my_configuration (if you want to use it) vim with_my # specify path to your mycfg (if you want to use it)
#+end_src #+end_src
It's also convenient to put =with_my= somewhere in your system path so you can run it from anywhere. It's also convenient to put =with_my= somewhere in your system path so you can run it from anywhere.
@ -113,8 +113,8 @@ or, set up as ~mypy.ini~ file:
#+begin_src #+begin_src
[mypy] [mypy]
mypy_path=/path/to/my_configuration_dir mypy_path=/path/to/mycfg_dir
#+end_src #+end_src
# TODO hmm, if package isn't using my_configuration then we don't really need it? # TODO hmm, if package isn't using mycfg then we don't really need it?

View file

@ -35,9 +35,9 @@ def run():
with_my = 'my_repo/with_my' with_my = 'my_repo/with_my'
copy('my_repo/with_my.example', with_my) copy('my_repo/with_my.example', with_my)
my_configuration_root = abspath('my_repo/my_configuration_template') mycfg_root = abspath('my_repo/mycfg_template')
# edit the config and set path to private configuration # edit the config and set path to private configuration
my = Path(with_my).read_text().replace("MY_CONFIGURATION_DIR = ''", "MY_CONFIGURATION_DIR = '{}'".format(my_configuration_root)) my = Path(with_my).read_text().replace("MYCFG_DIR = ''", "MYCFG_DIR = '{}'".format(mycfg_root))
Path(with_my).write_text(my) Path(with_my).write_text(my)
# #

View file

@ -13,23 +13,23 @@ Various thoughts on organizing
- not mypy/pylint friendly at all? - not mypy/pylint friendly at all?
- Second alternative: - Second alternative:
symlinks in my_configuration and direct import? symlinks in mycfg and direct import?
+ mypy/pylint friendly + mypy/pylint friendly
? keeping a symlink to model.py is not much worse than hardcoding path. so it's ok I guess ? keeping a symlink to model.py is not much worse than hardcoding path. so it's ok I guess
* Thoughts on organizing imports * Thoughts on organizing imports
- First way: - First way:
import my_configuration.hypexport_model as hypexport import mycfg.hypexport_model as hypexport
works, but my_configuration is scattered across the repository? works, but mycfg is scattered across the repository?
Second way: Second way:
from . import my_configuration? from . import mycfg?
doesn't seem to work with subpackages? doesn't seem to work with subpackages?
right, perhaps symlinking is a good idea after all?... right, perhaps symlinking is a good idea after all?...
Third way: Third way:
import my_configuration.repos.hypexport.model as hypexport import mycfg.repos.hypexport.model as hypexport
works, but MYPYPATH doesn't seem to be happy... works, but MYPYPATH doesn't seem to be happy...
ok, --namespace-packages solves it.. ok, --namespace-packages solves it..

View file

@ -1,13 +1,7 @@
# TODO just eval setup file to populate paths etc?
# TODO note sure if it would
# TODO maybe just import everything?
# TODO how to make it mypy friendly? maybe defensive import? or mypy config? or interface file? # TODO how to make it mypy friendly? maybe defensive import? or mypy config? or interface file?
try: try:
import my_configuration import mycfg
paths = my_configuration.paths # type: ignore
except ImportError: except ImportError:
import warnings import warnings
warnings.warn("my_configuration package isn't found! That might result in issues") warnings.warn("mycfg package isn't found! That might result in issues")
paths = None # type: ignore

View file

@ -14,7 +14,7 @@ from kython import dictify
# TODO vendorize in my. pkg? It's quite handy... # TODO vendorize in my. pkg? It's quite handy...
from kython.klogging import LazyLogger from kython.klogging import LazyLogger
from my_configuration import paths from mycfg import paths
logger = LazyLogger('bluemaestro', level=logging.DEBUG) logger = LazyLogger('bluemaestro', level=logging.DEBUG)

View file

@ -8,7 +8,7 @@ from kython import listify
from kython.org import parse_org_date from kython.org import parse_org_date
from kython.kerror import Res, echain from kython.kerror import Res, echain
from my_configuration import paths from mycfg import paths
import pandas as pd # type: ignore import pandas as pd # type: ignore

View file

@ -1,7 +1,7 @@
from typing import Callable, Union, List from typing import Callable, Union, List
from my_configuration import paths from mycfg import paths
from my_configuration.repos.kobuddy.src.kobuddy import * from mycfg.repos.kobuddy.src.kobuddy import *
set_databases(paths.kobuddy.export_dir) set_databases(paths.kobuddy.export_dir)

View file

@ -4,7 +4,7 @@ import re
from typing import Tuple, Iterator, List, Union from typing import Tuple, Iterator, List, Union
from my_configuration.holidays_data import HOLIDAYS_DATA from mycfg.holidays_data import HOLIDAYS_DATA
# pip3 install workalendar # pip3 install workalendar

View file

@ -8,8 +8,8 @@ import pytz
from ..common import get_files from ..common import get_files
from my_configuration import paths from mycfg import paths
import my_configuration.repos.ghexport.model as ghexport import mycfg.repos.ghexport.model as ghexport
def get_logger(): def get_logger():

View file

@ -3,8 +3,8 @@ from shutil import rmtree
from tempfile import TemporaryDirectory from tempfile import TemporaryDirectory
from typing import Iterator, Union from typing import Iterator, Union
import my_configuration.repos.fbmessengerexport.model as messenger import mycfg.repos.fbmessengerexport.model as messenger
from my_configuration import paths from mycfg import paths
def _dump_helper(model: messenger.Model, tdir: Path) -> None: def _dump_helper(model: messenger.Model, tdir: Path) -> None:

View file

@ -1,7 +1,8 @@
from . import paths
from .common import listify from .common import listify
from ._rss import Subscription from ._rss import Subscription
from mycfg import paths
import json import json
from pathlib import Path from pathlib import Path
from typing import Dict, List from typing import Dict, List

View file

@ -1,7 +1,8 @@
from . import paths
from .common import listify from .common import listify
from ._rss import Subscription from ._rss import Subscription
from mycfg import paths
import json import json
from pathlib import Path from pathlib import Path
from typing import Dict, List from typing import Dict, List

View file

@ -22,7 +22,7 @@ def get_logger():
def _get_exports() -> List[Path]: def _get_exports() -> List[Path]:
from my_configuration import paths from mycfg import paths
return get_files(paths.foursquare.export_path, '*.json') return get_files(paths.foursquare.export_path, '*.json')

View file

@ -6,9 +6,9 @@ from .common import group_by_key, the, cproperty, PathIsh, import_file
try: try:
# TODO might be worth having a special mode for type checking with and without my_configuration? # TODO might be worth having a special mode for type checking with and without mycfg?
# TODO could somehow use typing.TYPE_CHECKING for that? # TODO could somehow use typing.TYPE_CHECKING for that?
import my_configuration.repos.hypexport.model as hypexport import mycfg.repos.hypexport.model as hypexport
Highlight = hypexport.Highlight Highlight = hypexport.Highlight
Model = hypexport.Model Model = hypexport.Model
except: except:
@ -26,7 +26,7 @@ class Config(NamedTuple):
if ep is not None: if ep is not None:
return Path(ep) return Path(ep)
else: else:
from my_configuration import paths from mycfg import paths
return Path(paths.hypothesis.export_path) return Path(paths.hypothesis.export_path)
@property @property
@ -37,7 +37,7 @@ class Config(NamedTuple):
else: else:
global Model global Model
global Highlight global Highlight
import my_configuration.repos.hypexport.model as hypexport import mycfg.repos.hypexport.model as hypexport
# TODO a bit hacky.. not sure how to make it both mypy and runtime safe.. # TODO a bit hacky.. not sure how to make it both mypy and runtime safe..
Model = hypexport.Model Model = hypexport.Model
Highlight = hypexport.Highlight Highlight = hypexport.Highlight
@ -52,7 +52,7 @@ def configure(*, export_path: Optional[PathIsh]=None, hypexport_path: Optional[P
hypexport_path_=hypexport_path, hypexport_path_=hypexport_path,
) )
# TODO for the purposes of mypy, try importing my_configuration anyway? # TODO for the purposes of mypy, try importing mycfg anyway?
# return type for this method as well # return type for this method as well
# TODO check if it works at runtime.. # TODO check if it works at runtime..
def get_model() -> Model: def get_model() -> Model:

View file

@ -3,7 +3,7 @@ Uses instapaper API data export JSON file.
Set via Set via
- ~configure~ method - ~configure~ method
- or in ~my_configuration.instpaper.export_path~ - or in ~mycfg.instpaper.export_path~
TODO upload my exporter script to github.. TODO upload my exporter script to github..
""" """
@ -29,8 +29,8 @@ def _get_files():
# TODO use helper method from common to get json[s]? # TODO use helper method from common to get json[s]?
export_path = _export_path export_path = _export_path
if export_path is None: if export_path is None:
# fallback to my_configuration # fallback to mycfg
# TODO import my_configuration? # TODO import mycfg?
from . import paths from . import paths
export_path = paths.instapaper.export_path export_path = paths.instapaper.export_path
return list(sorted(Path(export_path).glob('*.json'))) return list(sorted(Path(export_path).glob('*.json')))

View file

@ -7,7 +7,7 @@ from pathlib import Path
import logging import logging
import pytz import pytz
from my_configuration import paths from mycfg import paths
BDIR = paths.jawbone.export_dir BDIR = paths.jawbone.export_dir

View file

@ -87,7 +87,7 @@ def iter_useful(data_file: str):
# TODO <<< hmm. these files do contain deep and light sleep?? # TODO <<< hmm. these files do contain deep and light sleep??
# also steps stats?? # also steps stats??
from my_configuration import paths from mycfg import paths
p = paths.jawbone.export_dir / 'old_csv' p = paths.jawbone.export_dir / 'old_csv'
# TODO with_my? # TODO with_my?

View file

@ -7,7 +7,7 @@ import json
import pytz import pytz
from my_configuration import paths from mycfg import paths
# TODO Json type? # TODO Json type?
# TODO memoised properties? # TODO memoised properties?

View file

@ -6,8 +6,7 @@ from typing import Iterator, List, NamedTuple
from ..common import get_files from ..common import get_files
# TODO eh. rename to my_cfg? easier to type from mycfg import paths
from my_configuration import paths
def _get_last(): def _get_last():
return max(get_files(paths.imdb.export_path, glob='*.csv')) return max(get_files(paths.imdb.export_path, glob='*.csv'))

View file

@ -8,7 +8,7 @@ from kython.kompress import open as kopen
from ..common import get_files from ..common import get_files
from my_configuration import paths from mycfg import paths
def _get_last_takeout(): def _get_last_takeout():

View file

@ -1,12 +1,12 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
from . import paths
from .common import import_file from .common import import_file
from pathlib import Path from pathlib import Path
# path to pdfannots (https://github.com/0xabu/pdfannots) # path to pdfannots (https://github.com/0xabu/pdfannots)
pdfannots = import_file(paths.pdfs.pdfannots_py) import mycfg.repos.pdfannots.pdfannots as pdfannots
from mycfg import paths
from datetime import datetime from datetime import datetime

View file

@ -41,7 +41,7 @@ class Article(NamedTuple):
# TODO add tags? # TODO add tags?
# TODO integrate with my_configuration # TODO integrate with mycfg
def get_articles(json_path: Path) -> Sequence[Article]: def get_articles(json_path: Path) -> Sequence[Article]:
import json import json
raw = json.loads(json_path.read_text())['list'] raw = json.loads(json_path.read_text())['list']

View file

@ -2,8 +2,8 @@
from pathlib import Path, PosixPath from pathlib import Path, PosixPath
from typing import List, Sequence, Mapping from typing import List, Sequence, Mapping
from my_configuration import paths from mycfg import paths
import my_configuration.repos.rexport.model as rexport import mycfg.repos.rexport.model as rexport
# TODO Move this to kython.kompress? # TODO Move this to kython.kompress?

View file

@ -9,7 +9,7 @@ from .common import get_files
# TODO get rid of it # TODO get rid of it
from kython import group_by_cmp # type: ignore from kython import group_by_cmp # type: ignore
from my_configuration import paths from mycfg import paths
def get_logger(): def get_logger():
@ -17,11 +17,11 @@ def get_logger():
def _get_exports() -> List[Path]: def _get_exports() -> List[Path]:
from my_configuration import paths from mycfg import paths
return get_files(paths.rescuetime.export_path, '*.json') return get_files(paths.rescuetime.export_path, '*.json')
import my_configuration.repos.rescuexport.model as rescuexport import mycfg.repos.rescuexport.model as rescuexport
Model = rescuexport.Model Model = rescuexport.Model

View file

@ -5,7 +5,7 @@ See https://help.twitter.com/en/managing-your-account/how-to-download-your-twitt
Expects path to be set Expects path to be set
- via ~configure~ (before calling anything else) - via ~configure~ (before calling anything else)
- or in ~my_configuration.twitter.export_path~ - or in ~mycfg.twitter.export_path~
""" """
@ -30,8 +30,8 @@ def configure(*, export_path: Optional[PathIsh]=None) -> None:
def _get_export() -> Path: def _get_export() -> Path:
export_path = _export_path export_path = _export_path
if export_path is None: if export_path is None:
# fallback to my_configuration # fallback to mycfg
from . import paths from mycfg import paths
export_path = paths.twitter.export_path export_path = paths.twitter.export_path
p = Path(export_path) p = Path(export_path)
if p.is_dir(): if p.is_dir():

View file

@ -2,7 +2,7 @@ from datetime import datetime
import json import json
from typing import NamedTuple, Iterator, Dict, Union, Sequence, Optional from typing import NamedTuple, Iterator, Dict, Union, Sequence, Optional
from my_configuration import paths from mycfg import paths
class Favorite(NamedTuple): class Favorite(NamedTuple):

View file

@ -13,7 +13,7 @@ Feel free to use your preferred way of managing these packages otherwise.
# can be empty if you're not planning to use modules that use private configuration # can be empty if you're not planning to use modules that use private configuration
# otherwise see readme to learn how to set it # otherwise see readme to learn how to set it
MY_CONFIGURATION_DIR = '' MYCFG_DIR = ''
###### ######
@ -39,10 +39,10 @@ if __name__ == '__main__':
val = val + ':' + path val = val + ':' + path
os.environ[envvar] = val os.environ[envvar] = val
# TODO wonder why py.typed file in my_configuration didn't help? # TODO wonder why py.typed file in mycfg didn't help?
for v in ['MYPYPATH', 'PYTHONPATH']: for v in ['MYPYPATH', 'PYTHONPATH']:
upd(v, MY_DIR) upd(v, MY_DIR)
upd(v, MY_CONFIGURATION_DIR) upd(v, MYCFG_DIR)
rest = sys.argv[1:] rest = sys.argv[1:]
os.execvp(rest[0], rest) os.execvp(rest[0], rest)