diff --git a/README.md b/README.md index 601f2f9..8d02e6a 100644 --- a/README.md +++ b/README.md @@ -34,9 +34,13 @@ pipx install pricehist - **`gnucash-sql`**: [GnuCash](https://www.gnucash.org/) SQL - **`ledger`**: [Ledger](https://www.ledger-cli.org/) and [hledger](https://hledger.org/) -## Examples +## Usage -Show usage information: +The following sections demonstrate different uses of `pricehist`. + +### Online help + +The `-h` option will show usage information: ```bash pricehist -h @@ -58,7 +62,7 @@ commands: fetch fetch prices ``` -Show usage information for the `fetch` command: +Here's the usage information for the `fetch` command: ``` pricehist fetch -h @@ -94,28 +98,48 @@ optional arguments: --fmt-csvdelim CHAR field delimiter for CSV output (default: ',') ``` -Fetch prices after 2021-01-04, ending 2021-01-15, as CSV: +### Fetching prices + +Fetch prices by choosing a source, a pair and, optionally, a time interval: ```bash -pricehist fetch ecb EUR/AUD -sx 2021-01-04 -e 2021-01-15 -o csv +pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 ``` ``` date,base,quote,amount,source,type +2021-01-04,EUR,AUD,1.5928,ecb,reference 2021-01-05,EUR,AUD,1.5927,ecb,reference 2021-01-06,EUR,AUD,1.5824,ecb,reference 2021-01-07,EUR,AUD,1.5836,ecb,reference 2021-01-08,EUR,AUD,1.5758,ecb,reference -2021-01-11,EUR,AUD,1.5783,ecb,reference -2021-01-12,EUR,AUD,1.5742,ecb,reference -2021-01-13,EUR,AUD,1.5734,ecb,reference -2021-01-14,EUR,AUD,1.5642,ecb,reference -2021-01-15,EUR,AUD,1.568,ecb,reference ``` -In Ledger format: +The default output format is CSV, which is suitable for use in spreadsheets and +with other tools. For example, on the command line using +[gnuplot](http://www.gnuplot.info/) we can chart Bitcoin prices: ```bash -pricehist fetch ecb EUR/AUD -s 2021-01-01 -o ledger | head +pricehist fetch coindesk BTC/USD -s 2021-01-01 | \ + cut -d, -f1,4 | \ + sed 1d | \ + gnuplot -p -e ' + set datafile separator ","; + set xdata time; + set timefmt "%Y-%m-%d"; + set format x "%b\n%Y"; + plot "/dev/stdin" using 1:2 with lines title "BTC/USD" + ' +``` + +![BTC/USD prices](example-gnuplot.png) + +### Choosing and customizing the output format + +You can choose a different output format: `ledger`, `beancount` or +`gnucash-sql`. + +```bash +pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger ``` ``` P 2021-01-04 00:00:00 EUR 1.5928 AUD @@ -123,14 +147,28 @@ P 2021-01-05 00:00:00 EUR 1.5927 AUD P 2021-01-06 00:00:00 EUR 1.5824 AUD P 2021-01-07 00:00:00 EUR 1.5836 AUD P 2021-01-08 00:00:00 EUR 1.5758 AUD -P 2021-01-11 00:00:00 EUR 1.5783 AUD -P 2021-01-12 00:00:00 EUR 1.5742 AUD -P 2021-01-13 00:00:00 EUR 1.5734 AUD -P 2021-01-14 00:00:00 EUR 1.5642 AUD -P 2021-01-15 00:00:00 EUR 1.568 AUD ``` -Generate SQL for a GnuCash database and apply it immediately: +The formatting options let you control certain details of the output. The +following example removes the time of day (to make it suitable for hledger) and +makes a few other adjustments. + +```bash +pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger \ + --fmt-time '' --fmt-datesep / --fmt-base € --fmt-quote $ --fmt-symbol left +``` +``` +P 2021/01/04 € $1.5928 +P 2021/01/05 € $1.5927 +P 2021/01/06 € $1.5824 +P 2021/01/07 € $1.5836 +P 2021/01/08 € $1.5758 +``` + +### GnuCash SQL output + +You can generate SQL for a GnuCash database and apply it immediately with one +of the following commands: ```bash pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | sqlite3 Accounts.gnucash @@ -138,12 +176,197 @@ pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | mysql -u username -p pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | psql -U username -d databasename -v ON_ERROR_STOP=1 ``` -## Design choices +Beware that the GnuCash project itself does not support integration at the +database level, so there is a risk that the SQL generated by `pricehist` will +be ineffective or even damaging for some new version of GnuCash. -To keep things simple, at least for now, `pricehist` provides only univariate -time series of daily historical prices. It doesn't provide other types of -market, financial or economic data, real-time prices, or other temporal -resolutions. Multiple or multivariate series require multiple invocations. +In practice, this strategy has been used successfully by other projects. +Reviewing the SQL and keeping regular database backups is recommended. + +### Source information + +Some basic information about each source is available: + +```bash +pricehist source ecb +``` +``` +ID : ecb +Name : European Central Bank +Description : European Central Bank Euro foreign exchange reference rates +URL : https://www.ecb.europa.eu/stats/exchange/eurofxref/html/index.en.html +Start : 1999-01-04 +Types : reference +``` + +Symbol information can be listed for most sources, either as full pairs or as +separate base and quote symbols that work in particular combinations: + +```bash +pricehist source ecb --symbols +``` +``` +EUR/AUD Euro against Australian Dollar +EUR/BGN Euro against Bulgarian Lev +EUR/BRL Euro against Brazilian Real +EUR/CAD Euro against Canadian Dollar +EUR/CHF Euro against Swiss Franc +... +``` + +It may also be possible to search for symbols: + +```bash +pricehist source alphavantage --search Tesla +``` +``` +TL0.DEX Tesla, Equity, XETRA, EUR +TL0.FRK Tesla, Equity, Frankfurt, EUR +TSLA34.SAO Tesla, Equity, Brazil/Sao Paolo, BRL +TSLA Tesla Inc, Equity, United States, USD +TXLZF Tesla Exploration Ltd, Equity, United States, USD +``` + +### Inspecting source interactions + +You can see extra information, including `curl` commands to reproduce each +request to a source, by adding the verbose option (`--verbose` or `-vvv`): + +```bash +pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv +``` +``` +DEBUG Began pricehist run at 2021-08-12 14:38:26.630357. +DEBUG Starting new HTTPS connection (1): api.coindesk.com:443 +DEBUG https://api.coindesk.com:443 "GET /v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05 HTTP/1.1" 200 319 +DEBUG curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.25.1' --compressed 'https://api.coindesk.com/v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05' +DEBUG Available data covers the interval [2021-01-01--2021-01-05], as requested. +date,base,quote,amount,source,type +2021-01-01,BTC,USD,29391.775,coindesk,close +2021-01-02,BTC,USD,32198.48,coindesk,close +2021-01-03,BTC,USD,33033.62,coindesk,close +2021-01-04,BTC,USD,32017.565,coindesk,close +2021-01-05,BTC,USD,34035.0067,coindesk,close +DEBUG Ended pricehist run at 2021-08-12 14:38:26.709428. +``` + +You can run a logged `curl` command to see exactly what data is returned by the +source: + +```bash +pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv 2>&1 \ + | grep 'DEBUG curl' | sed 's/^DEBUG //' | bash | jq . +``` +```json +{ + "bpi": { + "2021-01-01": 29391.775, + "2021-01-02": 32198.48, + "2021-01-03": 33033.62, + "2021-01-04": 32017.565, + "2021-01-05": 34035.0067 + }, + "disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index. BPI value data returned as USD.", + "time": { + "updated": "Jan 6, 2021 00:03:00 UTC", + "updatedISO": "2021-01-06T00:03:00+00:00" + } +} +``` + +### Use as a library + +While not yet optimized for use as a library, you may find `pricehist`'s source +classes useful in your own scripts: + +```python +$ python +Python 3.9.6 (default, Jun 30 2021, 10:22:16) +[GCC 11.1.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> from pricehist.series import Series +>>> from pricehist.sources.ecb import ECB +>>> series = ECB().fetch(Series("EUR", "AUD", "reference", "2021-01-04", "2021-01-08")) +>>> series.prices +[Price(date='2021-01-04', amount=Decimal('1.5928')), Price(date='2021-01-05', amount=Decimal('1.5927')), Price(date='2021-01-06', amount=Decimal('1.5824')), Price(date='2021-01-07', amount=Decimal('1.5836')), Price(date='2021-01-08', amount=Decimal('1.5758'))] +``` + +A subclass of `pricehist.exceptions.SourceError` will be raised for any error. + +## Terminology + +A **source** is the upstream service that can provide a series of prices. + +Each **series** of prices is for one pair and price type. + +The [**pair**](https://en.wikipedia.org/wiki/Currency_pair) is made up of a +base and a quote, each given as a symbol. Sometimes you will give the base +only, and the quote will be determined with information from the source. The +available pairs, the symbols used in them and the available price types all +depend on the particular source used. + +The **base** is the currency or commodity being valued. Each price expresses +the value of one unit of the base. + +The **quote** is the unit used to express the value of the base. + +A **symbol** is a code or abbreviation for a currency or commodity. + +The **prices** in a series each have a date and an amount. + +The **amount** is the number of units of the quote that are equal to one unit +of the base. + +Consider the following command: + +```bash +pricehist fetch coindesk BTC/USD --type close +``` + +- **`coindesk`** is the ID of the CoinDesk Bitcoin Price Index source. +- **`BTC`** is the symbol for Bitcoin. +- **`USD`** is the symbol for the United States Dollar. +- **`BTC/USD`** is the pair Bitcoin against United States Dollar. +- **`close`** is the price type for last price of each day. + +A BTC/USD price of the amount 29,391.775 can be written as +"BTC/USD = 29391.775" or "BTC 29391.775 USD", and means that one Bitcoin is +worth 29,391.775 United States Dollars. + +## Initial design choices + +To keep things simple, `pricehist` provides only univariate time series of +daily historical prices. It doesn't provide other types of market, financial or +economic data, real-time prices, or other temporal resolutions. Multiple or +multivariate series require multiple invocations. + +## Future potential features + +While the following features are relatively low value or beyond `pricehist`'s +primary use case, they could be built upon the foundation it has established. + +- **Time of day**: Sources sometimes provide specific times for each day's + high/low prices and these could be preserved for output. This would require + changes to how dates are handled internally, clarification of time zone + handling and extension of the time formatting option. +- **Alternate resolutions**: Some sources can provide higher or lower + resolution data, such as hourly or weekly. These could be supported where + available. For other cases an option could be provided for downsampling data + before output. +- **Real-time prices**: These generally come from different soruce endpoints + than the historical data. Real-time prices will usually have a different + price type, such as `last`, `bid` or `ask`. Support for real-time prices + would allow adding sources that don't provide historical data. Start and end + times are irrelevant when requesting real-time prices. A "follow" option + could continuously poll for new prices. +- **Related non-price data**: Trading volume, spreads, split and dividend + events and other related data could be supported. The base/quote/type model + used for prices would work for some of this. Other things may require + extending the model. +- **Multivariate series**: Would allow, for example, fetching + high/low/open/close prices in a single invocation. +- **`format` command**: A command for rewriting existing CSV data into one of + the other output formats. ## Alternatives diff --git a/example-gnuplot.png b/example-gnuplot.png new file mode 100644 index 0000000..f73ca28 Binary files /dev/null and b/example-gnuplot.png differ