383 lines
14 KiB
Markdown
383 lines
14 KiB
Markdown
# pricehist
|
||
|
||
A command-line tool for fetching and formatting historical price data, with
|
||
support for multiple data sources and output formats.
|
||
|
||
[](https://gitlab.com/chrisberkhout/pricehist/-/commits/master)
|
||
[](https://gitlab.com/chrisberkhout/pricehist/-/commits/master)
|
||
[](https://badge.fury.io/py/pricehist)
|
||
[](https://pepy.tech/project/pricehist)
|
||
[](https://gitlab.com/chrisberkhout/pricehist/-/blob/master/LICENSE)
|
||
[](https://github.com/psf/black)
|
||
|
||
## Installation
|
||
|
||
Install via [pip](https://pip.pypa.io/en/stable/) or
|
||
[pipx](https://pypa.github.io/pipx/):
|
||
|
||
```bash
|
||
pipx install pricehist
|
||
```
|
||
|
||
## Sources
|
||
|
||
- **`alphavantage`**: [Alpha Vantage](https://www.alphavantage.co/)
|
||
- **`coindesk`**: [CoinDesk Bitcoin Price Index](https://www.coindesk.com/coindesk-api)
|
||
- **`coinmarketcap`**: [CoinMarketCap](https://coinmarketcap.com/)
|
||
- **`ecb`**: [European Central Bank Euro foreign exchange reference rates](https://www.ecb.europa.eu/stats/exchange/eurofxref/html/index.en.html)
|
||
- **`yahoo`**: [Yahoo! Finance](https://finance.yahoo.com/)
|
||
|
||
## Output formats
|
||
|
||
- **`beancount`**: [Beancount](http://furius.ca/beancount/)
|
||
- **`csv`**: [Comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)
|
||
- **`gnucash-sql`**: [GnuCash](https://www.gnucash.org/) SQL
|
||
- **`ledger`**: [Ledger](https://www.ledger-cli.org/) and [hledger](https://hledger.org/)
|
||
|
||
## Usage
|
||
|
||
The following sections demonstrate different uses of `pricehist`.
|
||
|
||
### Online help
|
||
|
||
The `-h` option will show usage information:
|
||
|
||
```bash
|
||
pricehist -h
|
||
```
|
||
```
|
||
usage: pricehist [-h] [--version] [-vvv] {sources,source,fetch} ...
|
||
|
||
Fetch historical price data
|
||
|
||
optional arguments:
|
||
-h, --help show this help message and exit
|
||
--version show version information
|
||
-vvv, --verbose show all log messages
|
||
|
||
commands:
|
||
{sources,source,fetch}
|
||
sources list sources
|
||
source show source details
|
||
fetch fetch prices
|
||
```
|
||
|
||
Here's the usage information for the `fetch` command:
|
||
|
||
```
|
||
pricehist fetch -h
|
||
```
|
||
```
|
||
usage: pricehist fetch SOURCE PAIR [-h] [-vvv] [-t TYPE] [-s DATE | -sx DATE] [-e DATE | -ex DATE]
|
||
[-o beancount|csv|gnucash-sql|ledger] [--invert] [--quantize INT]
|
||
[--fmt-base SYM] [--fmt-quote SYM] [--fmt-time TIME] [--fmt-decimal CHAR] [--fmt-thousands CHAR]
|
||
[--fmt-symbol rightspace|right|leftspace|left] [--fmt-datesep CHAR] [--fmt-csvdelim CHAR]
|
||
|
||
positional arguments:
|
||
SOURCE the source identifier
|
||
PAIR pair, usually BASE/QUOTE, e.g. BTC/USD
|
||
|
||
optional arguments:
|
||
-h, --help show this help message and exit
|
||
-vvv, --verbose show all log messages
|
||
-t TYPE, --type TYPE price type, e.g. close
|
||
-s DATE, --start DATE start date, inclusive (default: source start)
|
||
-sx DATE, --startx DATE start date, exclusive
|
||
-e DATE, --end DATE end date, inclusive (default: today)
|
||
-ex DATE, --endx DATE end date, exclusive
|
||
-o FMT, --output FMT output format (default: csv)
|
||
--invert invert the price, swapping base and quote
|
||
--quantize INT round to the given number of decimal places
|
||
--fmt-base SYM rename the base symbol in output
|
||
--fmt-quote SYM rename the quote symbol in output
|
||
--fmt-time TIME set a particular time of day in output (default: 00:00:00)
|
||
--fmt-decimal CHAR decimal point in output (default: '.')
|
||
--fmt-thousands CHAR thousands separator in output (default: '')
|
||
--fmt-symbol LOCATION commodity symbol placement in output (default: rightspace)
|
||
--fmt-datesep CHAR date separator in output (default: '-')
|
||
--fmt-csvdelim CHAR field delimiter for CSV output (default: ',')
|
||
```
|
||
|
||
### Fetching prices
|
||
|
||
Fetch prices by choosing a source, a pair and, optionally, a time interval:
|
||
|
||
```bash
|
||
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08
|
||
```
|
||
```
|
||
date,base,quote,amount,source,type
|
||
2021-01-04,EUR,AUD,1.5928,ecb,reference
|
||
2021-01-05,EUR,AUD,1.5927,ecb,reference
|
||
2021-01-06,EUR,AUD,1.5824,ecb,reference
|
||
2021-01-07,EUR,AUD,1.5836,ecb,reference
|
||
2021-01-08,EUR,AUD,1.5758,ecb,reference
|
||
```
|
||
|
||
The default output format is CSV, which is suitable for use in spreadsheets and
|
||
with other tools. For example, on the command line using
|
||
[gnuplot](http://www.gnuplot.info/) we can chart Bitcoin prices:
|
||
|
||
```bash
|
||
pricehist fetch coindesk BTC/USD -s 2021-01-01 | \
|
||
cut -d, -f1,4 | \
|
||
sed 1d | \
|
||
gnuplot -p -e '
|
||
set datafile separator ",";
|
||
set xdata time;
|
||
set timefmt "%Y-%m-%d";
|
||
set format x "%b\n%Y";
|
||
plot "/dev/stdin" using 1:2 with lines title "BTC/USD"
|
||
'
|
||
```
|
||
|
||
|
||
|
||
### Choosing and customizing the output format
|
||
|
||
You can choose a different output format: `ledger`, `beancount` or
|
||
`gnucash-sql`.
|
||
|
||
```bash
|
||
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger
|
||
```
|
||
```
|
||
P 2021-01-04 00:00:00 EUR 1.5928 AUD
|
||
P 2021-01-05 00:00:00 EUR 1.5927 AUD
|
||
P 2021-01-06 00:00:00 EUR 1.5824 AUD
|
||
P 2021-01-07 00:00:00 EUR 1.5836 AUD
|
||
P 2021-01-08 00:00:00 EUR 1.5758 AUD
|
||
```
|
||
|
||
The formatting options let you control certain details of the output. The
|
||
following example removes the time of day (to make it suitable for hledger) and
|
||
makes a few other adjustments.
|
||
|
||
```bash
|
||
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger \
|
||
--fmt-time '' --fmt-datesep / --fmt-base € --fmt-quote $ --fmt-symbol left
|
||
```
|
||
```
|
||
P 2021/01/04 € $1.5928
|
||
P 2021/01/05 € $1.5927
|
||
P 2021/01/06 € $1.5824
|
||
P 2021/01/07 € $1.5836
|
||
P 2021/01/08 € $1.5758
|
||
```
|
||
|
||
### GnuCash SQL output
|
||
|
||
You can generate SQL for a GnuCash database and apply it immediately with one
|
||
of the following commands:
|
||
|
||
```bash
|
||
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | sqlite3 Accounts.gnucash
|
||
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | mysql -u username -p -D databasename
|
||
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | psql -U username -d databasename -v ON_ERROR_STOP=1
|
||
```
|
||
|
||
Beware that the GnuCash project itself does not support integration at the
|
||
database level, so there is a risk that the SQL generated by `pricehist` will
|
||
be ineffective or even damaging for some new version of GnuCash.
|
||
|
||
In practice, this strategy has been used successfully by other projects.
|
||
Reviewing the SQL and keeping regular database backups is recommended.
|
||
|
||
### Source information
|
||
|
||
Some basic information about each source is available:
|
||
|
||
```bash
|
||
pricehist source ecb
|
||
```
|
||
```
|
||
ID : ecb
|
||
Name : European Central Bank
|
||
Description : European Central Bank Euro foreign exchange reference rates
|
||
URL : https://www.ecb.europa.eu/stats/exchange/eurofxref/html/index.en.html
|
||
Start : 1999-01-04
|
||
Types : reference
|
||
```
|
||
|
||
Symbol information can be listed for most sources, either as full pairs or as
|
||
separate base and quote symbols that work in particular combinations:
|
||
|
||
```bash
|
||
pricehist source ecb --symbols
|
||
```
|
||
```
|
||
EUR/AUD Euro against Australian Dollar
|
||
EUR/BGN Euro against Bulgarian Lev
|
||
EUR/BRL Euro against Brazilian Real
|
||
EUR/CAD Euro against Canadian Dollar
|
||
EUR/CHF Euro against Swiss Franc
|
||
...
|
||
```
|
||
|
||
It may also be possible to search for symbols:
|
||
|
||
```bash
|
||
pricehist source alphavantage --search Tesla
|
||
```
|
||
```
|
||
TL0.DEX Tesla, Equity, XETRA, EUR
|
||
TL0.FRK Tesla, Equity, Frankfurt, EUR
|
||
TSLA34.SAO Tesla, Equity, Brazil/Sao Paolo, BRL
|
||
TSLA Tesla Inc, Equity, United States, USD
|
||
TXLZF Tesla Exploration Ltd, Equity, United States, USD
|
||
```
|
||
|
||
### Inspecting source interactions
|
||
|
||
You can see extra information, including `curl` commands to reproduce each
|
||
request to a source, by adding the verbose option (`--verbose` or `-vvv`):
|
||
|
||
```bash
|
||
pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv
|
||
```
|
||
```
|
||
DEBUG Began pricehist run at 2021-08-12 14:38:26.630357.
|
||
DEBUG Starting new HTTPS connection (1): api.coindesk.com:443
|
||
DEBUG https://api.coindesk.com:443 "GET /v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05 HTTP/1.1" 200 319
|
||
DEBUG curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.25.1' --compressed 'https://api.coindesk.com/v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05'
|
||
DEBUG Available data covers the interval [2021-01-01--2021-01-05], as requested.
|
||
date,base,quote,amount,source,type
|
||
2021-01-01,BTC,USD,29391.775,coindesk,close
|
||
2021-01-02,BTC,USD,32198.48,coindesk,close
|
||
2021-01-03,BTC,USD,33033.62,coindesk,close
|
||
2021-01-04,BTC,USD,32017.565,coindesk,close
|
||
2021-01-05,BTC,USD,34035.0067,coindesk,close
|
||
DEBUG Ended pricehist run at 2021-08-12 14:38:26.709428.
|
||
```
|
||
|
||
You can run a logged `curl` command to see exactly what data is returned by the
|
||
source:
|
||
|
||
```bash
|
||
pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv 2>&1 \
|
||
| grep 'DEBUG curl' | sed 's/^DEBUG //' | bash | jq .
|
||
```
|
||
```json
|
||
{
|
||
"bpi": {
|
||
"2021-01-01": 29391.775,
|
||
"2021-01-02": 32198.48,
|
||
"2021-01-03": 33033.62,
|
||
"2021-01-04": 32017.565,
|
||
"2021-01-05": 34035.0067
|
||
},
|
||
"disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index. BPI value data returned as USD.",
|
||
"time": {
|
||
"updated": "Jan 6, 2021 00:03:00 UTC",
|
||
"updatedISO": "2021-01-06T00:03:00+00:00"
|
||
}
|
||
}
|
||
```
|
||
|
||
### Use as a library
|
||
|
||
While not yet optimized for use as a library, you may find `pricehist`'s source
|
||
classes useful in your own scripts:
|
||
|
||
```python
|
||
$ python
|
||
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
|
||
[GCC 11.1.0] on linux
|
||
Type "help", "copyright", "credits" or "license" for more information.
|
||
>>> from pricehist.series import Series
|
||
>>> from pricehist.sources.ecb import ECB
|
||
>>> series = ECB().fetch(Series("EUR", "AUD", "reference", "2021-01-04", "2021-01-08"))
|
||
>>> series.prices
|
||
[Price(date='2021-01-04', amount=Decimal('1.5928')), Price(date='2021-01-05', amount=Decimal('1.5927')), Price(date='2021-01-06', amount=Decimal('1.5824')), Price(date='2021-01-07', amount=Decimal('1.5836')), Price(date='2021-01-08', amount=Decimal('1.5758'))]
|
||
```
|
||
|
||
A subclass of `pricehist.exceptions.SourceError` will be raised for any error.
|
||
|
||
## Terminology
|
||
|
||
A **source** is the upstream service that can provide a series of prices.
|
||
|
||
Each **series** of prices is for one pair and price type.
|
||
|
||
The [**pair**](https://en.wikipedia.org/wiki/Currency_pair) is made up of a
|
||
base and a quote, each given as a symbol. Sometimes you will give the base
|
||
only, and the quote will be determined with information from the source. The
|
||
available pairs, the symbols used in them and the available price types all
|
||
depend on the particular source used.
|
||
|
||
The **base** is the currency or commodity being valued. Each price expresses
|
||
the value of one unit of the base.
|
||
|
||
The **quote** is the unit used to express the value of the base.
|
||
|
||
A **symbol** is a code or abbreviation for a currency or commodity.
|
||
|
||
The **prices** in a series each have a date and an amount.
|
||
|
||
The **amount** is the number of units of the quote that are equal to one unit
|
||
of the base.
|
||
|
||
Consider the following command:
|
||
|
||
```bash
|
||
pricehist fetch coindesk BTC/USD --type close
|
||
```
|
||
|
||
- **`coindesk`** is the ID of the CoinDesk Bitcoin Price Index source.
|
||
- **`BTC`** is the symbol for Bitcoin.
|
||
- **`USD`** is the symbol for the United States Dollar.
|
||
- **`BTC/USD`** is the pair Bitcoin against United States Dollar.
|
||
- **`close`** is the price type for last price of each day.
|
||
|
||
A BTC/USD price of the amount 29,391.775 can be written as
|
||
"BTC/USD = 29391.775" or "BTC 29391.775 USD", and means that one Bitcoin is
|
||
worth 29,391.775 United States Dollars.
|
||
|
||
## Initial design choices
|
||
|
||
To keep things simple, `pricehist` provides only univariate time series of
|
||
daily historical prices. It doesn't provide other types of market, financial or
|
||
economic data, real-time prices, or other temporal resolutions. Multiple or
|
||
multivariate series require multiple invocations.
|
||
|
||
## Future potential features
|
||
|
||
While the following features are relatively low value or beyond `pricehist`'s
|
||
primary use case, they could be built upon the foundation it has established.
|
||
|
||
- **Time of day**: Sources sometimes provide specific times for each day's
|
||
high/low prices and these could be preserved for output. This would require
|
||
changes to how dates are handled internally, clarification of time zone
|
||
handling and extension of the time formatting option.
|
||
- **Alternate resolutions**: Some sources can provide higher or lower
|
||
resolution data, such as hourly or weekly. These could be supported where
|
||
available. For other cases an option could be provided for downsampling data
|
||
before output.
|
||
- **Real-time prices**: These generally come from different soruce endpoints
|
||
than the historical data. Real-time prices will usually have a different
|
||
price type, such as `last`, `bid` or `ask`. Support for real-time prices
|
||
would allow adding sources that don't provide historical data. Start and end
|
||
times are irrelevant when requesting real-time prices. A "follow" option
|
||
could continuously poll for new prices.
|
||
- **Related non-price data**: Trading volume, spreads, split and dividend
|
||
events and other related data could be supported. The base/quote/type model
|
||
used for prices would work for some of this. Other things may require
|
||
extending the model.
|
||
- **Multivariate series**: Would allow, for example, fetching
|
||
high/low/open/close prices in a single invocation.
|
||
- **`format` command**: A command for rewriting existing CSV data into one of
|
||
the other output formats.
|
||
|
||
## Alternatives
|
||
|
||
Beancount's [`bean-price`](https://github.com/beancount/beanprice) tool fetches
|
||
prices and addresses other workflow concerns in a Beancount-specific manner,
|
||
generally requiring a Beancount file as input.
|
||
|
||
The [Piecash](https://piecash.readthedocs.io/) library is a pythonic interface
|
||
to GnuCash files stored in SQL which has a
|
||
[`Commodity.update_prices`](https://piecash.readthedocs.io/en/master/api/piecash.core.commodity.html?highlight=update_prices#piecash.core.commodity.Commodity.update_prices)
|
||
method for fetching historical prices.
|
||
The GnuCash wiki documents [wrapper scripts](https://wiki.gnucash.org/wiki/Stocks/get_prices)
|
||
for the [Finance::QuoteHist](https://metacpan.org/pod/Finance::QuoteHist) Perl
|
||
module.
|