Commit graph

86 commits

Author SHA1 Message Date
Dima Gerasimov
bf8af6c598 tox: try using uv for CI, should result in speedup
see https://github.com/karlicoss/HPI/issues/391
2024-09-23 01:22:16 +01:00
Dima Gerasimov
75639a3d5e tox: some prep for potentially using uv on CI instead of pip
see https://github.com/karlicoss/HPI/issues/391
2024-09-22 20:10:52 +01:00
Dima Gerasimov
b87d1c970a tests: move remaining tests from tests/ to my.tests, cleanup corresponding modules 2024-08-26 04:00:58 +01:00
Dima Gerasimov
a5643206a0 general: make time.tz.via_location user config lazy, move tests to my.tests package
also gets rid of the problematic reset_modules thingie
2024-08-26 04:00:58 +01:00
Dima Gerasimov
34593c032d tests: move more tests into core, more consistent tests running in tox 2024-08-07 01:08:39 +01:00
Dima Gerasimov
074e24c309 general: deprecate my.core.dataset and simplify tox file 2024-08-07 01:08:39 +01:00
Dima Gerasimov
fb8e9909a4 tests: simplify tests for my.core.serialize a bit and simplify tox file 2024-08-07 01:08:39 +01:00
Dima Gerasimov
0e6dd32afe ci: minor fixes after mypy update 2024-08-03 16:18:32 +01:00
Dima Gerasimov
87a8a7781b my.google.maps: intitial module for extracting placed data from Android app 2024-01-01 23:46:02 +00:00
karlicoss
09e0f66892 tox: disable --parallel flag in hpi module install
It's been so flaky it ends up taking more time to merge stuff. See https://github.com/karlicoss/HPI/issues/306
2023-11-19 19:18:19 +00:00
karlicoss
37643c098f tox: remove cat coverage index from tox, it's not very useful anyway 2023-11-10 23:11:54 +00:00
karlicoss
7b1cec9326 codeforces/topcode: move to top level and check in ci 2023-11-10 23:11:54 +00:00
karlicoss
ac5f71c68b my.jawbone: get rid of matplotlib import on top level 2023-11-10 02:10:30 +00:00
karlicoss
7631f1f2e4 monzo.monzoexport: initial module 2023-11-02 00:47:13 +00:00
Dima Gerasimov
0512488241 ci: sync configs to pymplate
- add python3.12
- add ruff
2023-10-06 02:24:01 +01:00
Dima Gerasimov
8cd74a9fc4 ci: attempt to use --parallel flag in tox 2023-10-02 01:27:49 +01:00
Dima Gerasimov
8addd2d58a new module: Harmonic app for Hackernews 2023-09-25 16:36:21 +01:00
Dima Gerasimov
f8cd31044e general: move reddit tests into my/tests + tweak my.core.cfg to be more reliable 2023-05-26 00:58:23 +01:00
Dima Gerasimov
9594caa1cd general: move most core tests inside my.core.tests package
- distributes tests alongside the package, might be convenient for package users
- removes some weird indirection (e.g. dummy test files improting tests from modules)
- makes the command line for tests cleaner (e.g. no need to remember to manually add files to tox.ini)
- tests automatically covered by mypy (so makes mypy runs cleaner and ultimately better coverage)

The (vague) convention is

- tests/somemodule.py -- testing my.core.somemodule, contains tests directly re
- tests/test_something.py -- testing a specific feature, e.g. test_get_files.py tests get_files methon only
2023-05-25 00:25:13 +01:00
seanbreckenridge
40de162fab
cli: add option to output locations to gpx files (#286)
* cli: add option to output locations to gpx files
2023-04-15 00:31:11 +01:00
Sean Breckenridge
f36bc6144b tox: use my.ip.all, sort hpi installs 2023-02-28 20:55:12 +00:00
Sean Breckenridge
435cb020f9 add example for denylist, update ci 2023-02-28 20:55:12 +00:00
seanbreckenridge
98b086f746
location fallback (#263)
see https://github.com/karlicoss/HPI/issues/262

* move home to fallback/via_home.py
* move via_ip to fallback
* add fallback model
* add stub via_ip file
* add fallback_locations for via_ip
* use protocol for locations
* estimate_from helper, via_home estimator, all.py
* via_home: add accuracy, cache history
* add datasources to gpslogger/google_takeout
* tz/via_location.py: update import to fallback
* denylist docs/installation instructions
* tz.via_location: let user customize cachew refresh time
* add via_ip.estimate_location using binary search
* use estimate_location in via_home.get_location
* tests: add gpslogger to location config stub
* tests: install tz related libs in test env
* tz: add regression test for broken windows dates

* vendorize bisect_left from python src
doesnt have a 'key' parameter till python3.10
2023-02-28 04:30:06 +00:00
Dima Gerasimov
07e7c62d02 general/ci: mypy check tests 2023-02-21 00:20:58 +00:00
Dima Gerasimov
c63177e186 general/ci: clean up mypy-misc pipeline, only exclude specific files instead
marked some module configs which aren't really ready for public use as type: ignore
2023-02-21 00:20:58 +00:00
Dima Gerasimov
5ac5636e7f core: better support for ad-hoc configs
properly reload/unload the relevant modules so hopefully no more weird hacks should be required

relevant
- https://github.com/karlicoss/promnesia/issues/340
- https://github.com/karlicoss/HPI/issues/46
2023-02-09 02:35:09 +00:00
karlicoss
11b6e51c90 ci: fix tox config
seems that after version 4.0 it's necessary to specify environments to run
previosly it was picking them up automatically
2023-01-31 00:31:54 +00:00
Dima Gerasimov
cef9b4c6d3 ci: try using --parallel install for mypy pipeline
`time tox -e mypy-misc` (removed the actual mypy call)

before (each module in a separate 'hpi install' command)
```
real	1m45.901s
user	1m19.555s
sys	0m5.491s
```

in a single 'hpi install' command (multiple modules)
```
real	1m31.252s
user	1m6.028s
sys	0m5.065s
```

single 'hpi install' command with --parallel
```
real	0m15.674s
user	0m50.986s
sys	0m3.249s
```
2022-06-06 09:49:15 +01:00
Dima Gerasimov
fd0c65d176 my.tinder: initial module for android databases 2022-06-04 17:16:28 +01:00
Dima Gerasimov
fd1a683d49 my.bumble: merge from all previous android exports 2022-06-02 14:21:21 +01:00
Dima Gerasimov
3faebdd629 core: add Protocol/TypedDict to compat 2022-06-02 14:21:21 +01:00
Dima Gerasimov
64a4782f0e core/ci: fix windows-specific issues
- use portable separators
- paths should be prepended with r' (so backwards slash isn't treated as escaping)
- sqlite connections should be closed (otherwise windows fails to remove the underlying db file)
- workaround for emojis via PYTHONUTF8=1 test for now
- make ZipPath portable
- properly use tox python environment everywhere

  this was causing issues on Windows
  e.g.
      WARNING: test command found but not installed in testenv
        cmd: C:\hostedtoolcache\windows\Python\3.9.12\x64\python3.EXE
2022-05-03 10:16:01 +01:00
Dima Gerasimov
637982a5ba ci: update ci configs
- add windows runner
- update actions versions
- other minor enhancements
2022-05-03 10:16:01 +01:00
seanbreckenridge
2cb836181b
location: add all.py, using takeout/gpslogger/ip (#237)
* location: add all.py, using takeout/gpslogger/ip, update docs
2022-04-26 21:11:35 +01:00
seanbreckenridge
9e5cd60ff2
browser: parse browser history using browserexport (#216)
* browser: parse browser history using browserexport

from seanbreckenridge/HPI module:
1fba8ccf2f/my/browser/export.py
2022-02-13 23:56:05 +00:00
Dima Gerasimov
5e9cc2a6a0 my.reddit: enable CI tests 2021-12-24 18:02:37 +00:00
Sean Breckenridge
8422c6e420
my.reddit: refactor into module that supports pushshift/gdpr (#179)
* initial pushshift/rexport merge implementation, using id for merging
* smarter module deprecation warning using regex
* add `RedditBase` from promnesia
* `import_source` helper for gracefully handing mixin data sources
2021-10-31 20:39:04 +00:00
Sean Breckenridge
4a04c09f31 docs: fix copy-paste errors/spelling mistakes 2021-07-10 10:56:23 +01:00
Sean Breckenridge
46198a6447
my.core.serialize: simplejson support, more types (#176)
* my.core.serialize: simplejson support, more types

I added a couple extra checks to the default function,
serializing datetime, dates and dataclasses (incase
orjson isn't installed)

(copied from below)

if orjson couldn't be imported, try simplejson
This is included for compatibility reasons because orjson
is rust-based and compiling on rarer architectures may not work
out of the box

as an example, I've been having issues getting it to install
on my phone (termux/android)

unlike the builtin JSON modue which serializes NamedTuples as lists
(even if you provide a default function), simplejson correctly
serializes namedtuples to dictionaries

this just gives another option to people, simplejson is pure python
so no one should have issues with that. orjson is still way faster,
so still preferable if its easy and theres a precompiled build
for your architecture (which there typically is)

If you're ever running this with simplejson installed and not orjson,
its pretty easy to tell as the JSON styling is different; orjson has
no spaces between tokens, simplejson puts spaces between tokens. e.g.

simplejson: {"a": 5, "b": 10}
orjson: {"a":5,"b":10}
2021-07-08 23:02:56 +01:00
Sean Breckenridge
821bc08a23
core/structure: help locate/extract gdpr exports (#175)
* core/structure: help locate/extract gdpr exports

* ci: add install-types to install stub packages
2021-07-08 00:44:55 +01:00
Sean Breckenridge
b64a11cc69
smscalls: allow multiple backup dirs (#172)
* smscalls: allow multiple backup dirs
* add smscalls to my.config, add test to CI
2021-05-14 01:35:36 +01:00
Dima Gerasimov
f09ca17560 core/sqlite: move tests to separate module, pickling during Pool.submit can't handle importing :( 2021-04-05 08:37:07 +01:00
Dima Gerasimov
ad177a1ccd my.pdfs: cleanup/refactor
- modernize:
  - add REQUIRES spec for pdfannots library
  - config dataclass/config stub
  - stats function
  - absolute my.core imports in anticipation of splitting core
- use 'paths' instead of 'roots' (better reflects the semantics), use get_files
  backward compatible via config migration
- properly run tests/mypy
2021-04-01 17:27:06 +01:00
Dima Gerasimov
29384aef44 my.goodreads: cleanup, rename from my.reading.goodrads & use proper pip dependency
related:
- https://github.com/karlicoss/HPI/issues/79
- 10d8cc86a1
2021-03-26 05:06:53 +00:00
Sean Breckenridge
eb26cf8633
my.core.serialize: orjson with additional default and _serialize hook (#140)
basic orjson serialize, json.dumps fallback

Lots of surrounding changes from this discussion:
0593c69056
2021-03-20 00:48:03 +00:00
Dima Gerasimov
a1a24ffbc3 my.coding.commits: more cleanup
Followup of https://github.com/karlicoss/HPI/pull/132

- add REQUIRES section
- use 'commits' config section & add proper schema
- use dedicated subdirectory for cache
2021-03-15 10:33:46 +00:00
Dima Gerasimov
ec8b0e9170 my.coding.commits: actually test on CI, add config stub 2021-03-15 10:33:46 +00:00
Dima Gerasimov
86497f9b13 new: basic arbtt module 2021-02-25 19:56:35 +00:00
Dima Gerasimov
da3c1c9b74 core/cachew: rely on ~/.cache for default cache path
- rely on appdirs for default cache path instead of hardcoded /var/tmp/cachew
  technically backwards incompatible, but no action needed
  you might want to clean /var/tmp/cachew after updating

- use default cache path (e.g. ~/.cache) by default
  see https://github.com/ActiveState/appdirs#some-example-output for more info
  *warning*: things will be cached by default now (used to be uncached before)

- treat cache_dir = None in the config
  *warning*: kind of backwards incompatible.. but again nothing disasterous
2021-02-21 19:51:58 +00:00
Dima Gerasimov
ad1cc71b0f readme: update 2021-02-20 01:39:55 +00:00