Dima Gerasimov
02dabe9f2b
my.twitter.archive: cleanup linting and use proper configuration via abstract class
2024-09-22 02:13:10 +01:00
Dima Gerasimov
239e6617fe
my.twitter.archive: deduplicate tweets based on id_str/created_at and raw tweet text
2024-09-22 02:13:10 +01:00
Dima Gerasimov
72cc8ff3ac
ruff: enable B warnings (mainly suppressed exceptions and unused variables)
2024-08-28 04:06:32 +01:00
Dima Gerasimov
d244c7cc4e
ruff: enable and fix C4 ruleset
2024-08-28 04:06:32 +01:00
Dima Gerasimov
a5c04e789a
twitter.archive: deduplicate results via json.dumps
...
this speeds up processing quite a bit, from 40s to 20s for me, plus removes tons of identical outputs
interesting enough, using raw object without json.dumps as key brings unique_everseen to crawl...
2023-10-24 01:54:30 +01:00
karlicoss
8c2d1c9463
general: use less explicit kompress boilerplate in modules
...
now get_files/kompress library can handle it transparently
2023-10-20 21:13:59 +01:00
Dima Gerasimov
dff31455f1
general: switch to make_logger in a few modules, use a bit more consistent logging, rely on default INFO level
2023-06-21 18:42:15 +01:00
Dima Gerasimov
fe88380499
general: switch to using native 3.8 versions for cached_property/Literal/Protocol instead of compat
2023-05-16 01:18:30 +01:00
Dima Gerasimov
c34656e8fb
general: update mypy config, seems that logs of type: ignore aren't necessary anymore
2023-05-16 01:18:30 +01:00
Dima Gerasimov
c63177e186
general/ci: clean up mypy-misc pipeline, only exclude specific files instead
...
marked some module configs which aren't really ready for public use as type: ignore
2023-02-21 00:20:58 +00:00
Dima Gerasimov
5f1d41fa52
my.twitter.archive: fix for newer format (tweets filename changed to tweets.js)
2022-10-19 00:06:23 +01:00
Dima Gerasimov
ca91be8154
twitter.archive: fix legacy config detection
...
apparently .name contains the parent module so previously it was throwing the exception instead
2022-10-19 00:06:23 +01:00
Dima Gerasimov
4e59a65f9a
core/general: move cached_property into compat, use standard implementation from python3.8
2022-05-31 14:08:50 +01:00
Dima Gerasimov
711157e0f5
my.twitter.archive: switch to zippath, add config section, better mypy coverage
2022-05-31 14:08:50 +01:00
Dima Gerasimov
bb6201bf2d
my.twitter.archive: expand entities in tweet text
2022-05-31 01:28:11 +01:00
Dima Gerasimov
1e2fc3bec7
twitter.archive: unescape stuff like </>
2022-05-31 01:28:11 +01:00
Dima Gerasimov
de7972be05
twitter: add permalink to Talon objects; extract shared method
2022-05-31 01:28:11 +01:00
Sean Breckenridge
62832a6756
twitter/archive: set default logger to warning
2022-02-09 23:18:24 +00:00
Sean Breckenridge
b6fa26b899
twitter/archive: update deprecated imports
2022-02-09 23:18:24 +00:00
Dima Gerasimov
b9852f45cf
twitter: use import_source and proper merging for tweets from different sources
...
+ use proper datetime_aware for created_at
2022-02-08 20:45:10 +00:00
Dima Gerasimov
afdf9d4334
twitter: initial talon module, processing data from Talon android app
2022-02-08 20:45:10 +00:00
Sean Breckenridge
5ecd4b4810
cleanup; remove unused imports
2021-04-02 08:38:06 +01:00
Dima Gerasimov
15789a4149
kyhton.kompress: move to core (with a fallback, used in promnesia)
2020-10-29 03:13:18 +01:00
Dima Gerasimov
fbaa8e0b44
core: add warnings helper to highlight warnings so they are more visible in the output
2020-09-27 17:47:30 +02:00
Dima Gerasimov
1cc4eb5d8d
core: add helper for computing stats; use it in modules
2020-06-04 22:19:34 +01:00
Dima Gerasimov
a267aeec5b
github: add config templates + docs
...
- ghexport: use export_path (export_dir is still supported)
2020-06-01 23:33:34 +01:00
Dima Gerasimov
f5267d05d7
my.twitter.archive: rename config (preserving bckwd compatibility for now)
2020-05-24 13:06:52 +01:00
Dima Gerasimov
b99b2f3cfa
core: add warning when get_files returns no files, my.twitter.archive: make more defensive in case of no archives
2020-05-24 12:51:23 +01:00
Dima Gerasimov
b7662378a2
docs: minor updates
2020-05-22 19:38:14 +01:00
Dima Gerasimov
03773a7b2c
twitter module: prettify top level twitter.all
2020-05-22 19:00:02 +01:00
Dima Gerasimov
e92ca215e3
Adapt takeout and twitter configs to the new pattern
...
Works fairly well so far?
2020-05-10 15:56:57 +01:00
Dima Gerasimov
8b8a85e8c3
kompress.kopen improvements
...
- tests
- uniform handling for bytes/str, always return utf8 str by default
2020-05-04 08:37:36 +01:00
Dima Gerasimov
51ae8601b4
Update docstrings and add links
2020-04-26 16:50:06 +01:00
Dima Gerasimov
96a850faf9
remove unnecessary methods from twitter provider
2020-04-20 08:38:01 +01:00
Dima Gerasimov
81986b0624
support likes from twint
2020-04-14 23:01:44 +01:00
Dima Gerasimov
30b6918a8d
unified view for twitter data
2020-04-14 22:05:47 +01:00
Dima Gerasimov
56b6ab9aaf
move twitter stuff to twitter subdir
2020-04-14 21:38:21 +01:00
