Dima Gerasimov
06084a8787
my.core.common: move warn_if_empty to my.core.utils.itertools, cleanup and add more tests
2024-08-16 10:22:29 +01:00
Dima Gerasimov
770dba5506
core.common: move away import related stuff to my.core.utils.imports
...
moving without backward compatibility, since it's extremely unlikely they are used for any external modules
in fact, unclear if these methods still have much value at all, but keeping for now just in case
2024-08-16 10:22:29 +01:00
Dima Gerasimov
66c08a6c80
core.common: move listify to core.utils.itertools, use better typing annotations for it
...
also some minor refactoring of my.rss
2024-08-16 10:22:29 +01:00
Dima Gerasimov
c64d7f5b67
core: cleanup itertool style helpers
...
- deprecate group_by_key, should use itertool.bucket instead
- move make_dict and ensure_unique to my.core.utils.itertools
2024-08-16 10:22:29 +01:00
Dima Gerasimov
973c4205df
core: cleanup deprecations, exclude from type checking and show runtime warnings
...
among affected things:
- core.common.assert_never
- core.common.cproperty
- core.common.isoparse
- core.common.mcachew
- core.common.the
- core.common.tzdatetime
- core.compat.sqlite_backup
2024-08-16 10:22:29 +01:00
Dima Gerasimov
a7439c7846
general: move assert_never to my.core.compat as it's in stdlib from 3.11
...
rely on typing-extensions for fallback
introducing typing-extensions dependency without fallback, should be ok since it's in the top 10 of popular packages
2024-08-16 10:22:29 +01:00
Dima Gerasimov
1317914bff
general: add 'destructive parsing' (kinda what we were doing in my.core.konsume) to my.experimental
...
also some cleanup for my.codeforces and my.topcoder
2024-08-12 13:24:28 +01:00
Dima Gerasimov
1e1e8d8494
my.topcoder: get rid of kjson in favor of using builtin dict methods
2024-08-12 13:24:28 +01:00
Dima Gerasimov
069264ce52
core.common: get rid of deprecated utcfromtimestamp
2024-08-10 17:46:30 +01:00
Dima Gerasimov
c69a0b43ba
my.vk.favorites: some minor cleanup
2024-08-10 17:46:30 +01:00
Dima Gerasimov
34593c032d
tests: move more tests into core, more consistent tests running in tox
2024-08-07 01:08:39 +01:00
Dima Gerasimov
074e24c309
general: deprecate my.core.dataset and simplify tox file
2024-08-07 01:08:39 +01:00
Dima Gerasimov
fb8e9909a4
tests: simplify tests for my.core.serialize a bit and simplify tox file
2024-08-07 01:08:39 +01:00
Dima Gerasimov
3aebc573e8
tests: use updated conftest from pymplate, this allows to run individual test modules properly
...
e.g. pytest --pyargs my.core.tests.test_get_files
2024-08-06 20:55:16 +01:00
Dima Gerasimov
b615ba10b1
ci: temporary suppress pandas mypy error in check_dateish
2024-08-05 23:35:24 +01:00
Dima Gerasimov
2c63fe25c0
my.twitter.android: get data from statues table rather that timeline_view
2024-08-05 23:35:24 +01:00
Dima Gerasimov
652ee9b875
fbmessenger.android: fix minor issue with processing thread participants
2024-08-03 19:01:51 +01:00
Dima Gerasimov
9e72672b4f
legacy google takeout: fix timezone localization
2024-08-03 16:50:09 +01:00
karlicoss
d5fccf1874
twitter.android: more comments on timeline types
2024-08-03 16:50:09 +01:00
Dima Gerasimov
0e6dd32afe
ci: minor fixes after mypy update
2024-08-03 16:18:32 +01:00
Dima Gerasimov
c9c0e19543
my.instagram.gdpr: fix for new format
2024-08-03 16:18:32 +01:00
seanbreckenridge
35dd5d82a0
smscalls: parse mms from smscalls export ( #370 )
...
* initial mms exploration
2024-06-05 22:03:03 +01:00
Dima Gerasimov
8a8a1ebb0e
my.tinder.android: better error handing and fix case with empty db
2024-04-03 20:13:40 +01:00
Dima Gerasimov
103ea2096e
my.coding.commits: fix for git repo discovery after fdfind v9
2024-03-13 00:46:18 +00:00
Dima Gerasimov
7236024c7a
my.twitter.android: better detection of own user id
2024-03-13 00:46:18 +00:00
Dima Gerasimov
87a8a7781b
my.google.maps: intitial module for extracting placed data from Android app
2024-01-01 23:46:02 +00:00
Sean Breckenridge
93e475795d
google takeout: support multiple locales
...
uses the known locales in google_takeout_parser
to determine the expected paths for each locale,
and performs a partial match on the paths to
detect and use match_structure
2023-12-31 18:57:30 +00:00
Dima Gerasimov
1b187b2c1b
whatsapp.android: expose all entities extracted from the db
2023-12-29 00:57:49 +00:00
Dima Gerasimov
3ec362fce9
fbmessenger.android: expose contacts
2023-12-28 18:13:16 +00:00
karlicoss
a0ce666024
my.youtube.takeout: fix exception handling
2023-12-28 00:25:05 +00:00
karlicoss
1c452b12d4
twitter.android: extract likes and own tweets as well
2023-12-28 00:12:39 +00:00
karlicoss
51209c547e
my.twitter.android: refactor into a proper module
...
for now only extracting bookmarks, will use it for some time and see how it goes
2023-12-24 00:49:07 +00:00
karlicoss
a4a7bc41b9
my.twitter.android: extract entities
2023-12-24 00:49:07 +00:00
karlicoss
3d75abafe9
my.twitter.android: some intial work on pasring sqlite databases from official Android app
2023-12-24 00:49:07 +00:00
Sean Breckenridge
224ba521e3
gpslogger: catch broken xml file error
2023-12-20 02:41:52 +00:00
Dima Gerasimov
a843407e40
core/compat: move fromisoformat to .core.compat module
2023-11-19 23:45:08 +00:00
Dima Gerasimov
bde43d6a7a
my.body.sleep: massive speedup for average temperature calculation
2023-11-11 00:42:49 +00:00
karlicoss
7b1cec9326
codeforces/topcode: move to top level and check in ci
2023-11-10 23:11:54 +00:00
karlicoss
657ce08ac8
fix mypy issues after mypy/libraries updates
2023-11-10 22:59:09 +00:00
karlicoss
996169aa29
time.tz.via_location: more consistent behaviour wrt caching
...
previously it was possible to cachew never properly initialize the cache because if you only queried some dates in the past
because we never made it to the end of _iter_tzs
also some minor cleanup
2023-11-10 22:59:09 +00:00
karlicoss
70bb9ed0c5
location.google_takeout_semantic: handle None visitConfidence
2023-11-10 02:10:30 +00:00
karlicoss
65c617ed94
my.emfit: add missing properties to fake data generator
2023-11-10 02:10:30 +00:00
karlicoss
ac5f71c68b
my.jawbone: get rid of matplotlib import on top level
2023-11-10 02:10:30 +00:00
karlicoss
33f8d867e2
my.browser.export: cleanup
...
- make logging INFO (default) -- otherwise it's too quiet during processing lots of databases
- can pass inputs cachew directly now
2023-11-07 21:24:56 +00:00
karlicoss
19353e996d
my.hackernews.harmonic: use orjson + add __hash__ for Saved object
...
plus some minor cleanup
2023-11-07 01:03:57 +00:00
karlicoss
4ac3bbb101
my.bumble.android: fix message deduplication
2023-11-07 01:03:57 +00:00
karlicoss
5630621ec1
my.pinboard: some cleanup
2023-11-06 23:10:00 +00:00
karlicoss
7631f1f2e4
monzo.monzoexport: initial module
2023-11-02 00:47:13 +00:00
karlicoss
105928238f
vk_messages_backup: some cleanup + switch to get_files
2023-11-02 00:43:10 +00:00
karlicoss
71cb66df5f
core: add helper for more_iterable to check that all types involved are hashable
...
Otherwise unique_everseen performance may degrade to quadratic rather than linear
For now hidden behind HPI_CHECK_UNIQUE_EVERSEEN flag
also switch some modules to use it
2023-10-31 01:02:17 +00:00
