fz0x1/pricehist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Expand README.

Browse source
This commit is contained in:
Chris Berkhout 2021-08-12 16:30:53 +02:00
parent c3a186fe49
commit a4c0a142e3
2 changed files with 246 additions and 23 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

269
README.md
View file

@ -34,9 +34,13 @@ pipx install pricehist
- **`gnucash-sql`**: [GnuCash](https://www.gnucash.org/) SQL - **`gnucash-sql`**: [GnuCash](https://www.gnucash.org/) SQL
- **`ledger`**: [Ledger](https://www.ledger-cli.org/) and [hledger](https://hledger.org/) - **`ledger`**: [Ledger](https://www.ledger-cli.org/) and [hledger](https://hledger.org/)
## Examples ## Usage
Show usage information: The following sections demonstrate different uses of `pricehist`.
### Online help
The `-h` option will show usage information:
```bash ```bash
pricehist -h pricehist -h
@ -58,7 +62,7 @@ commands:
fetch fetch prices fetch fetch prices
``` ```
Show usage information for the `fetch` command: Here's the usage information for the `fetch` command:
``` ```
pricehist fetch -h pricehist fetch -h
@ -94,28 +98,48 @@ optional arguments:
--fmt-csvdelim CHAR field delimiter for CSV output (default: ',') --fmt-csvdelim CHAR field delimiter for CSV output (default: ',')
``` ```
Fetch prices after 2021-01-04, ending 2021-01-15, as CSV: ### Fetching prices
Fetch prices by choosing a source, a pair and, optionally, a time interval:
```bash ```bash
pricehist fetch ecb EUR/AUD -sx 2021-01-04 -e 2021-01-15 -o csv pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08
``` ```
``` ```
date,base,quote,amount,source,type date,base,quote,amount,source,type
2021-01-04,EUR,AUD,1.5928,ecb,reference
2021-01-05,EUR,AUD,1.5927,ecb,reference 2021-01-05,EUR,AUD,1.5927,ecb,reference
2021-01-06,EUR,AUD,1.5824,ecb,reference 2021-01-06,EUR,AUD,1.5824,ecb,reference
2021-01-07,EUR,AUD,1.5836,ecb,reference 2021-01-07,EUR,AUD,1.5836,ecb,reference
2021-01-08,EUR,AUD,1.5758,ecb,reference 2021-01-08,EUR,AUD,1.5758,ecb,reference
2021-01-11,EUR,AUD,1.5783,ecb,reference
2021-01-12,EUR,AUD,1.5742,ecb,reference
2021-01-13,EUR,AUD,1.5734,ecb,reference
2021-01-14,EUR,AUD,1.5642,ecb,reference
2021-01-15,EUR,AUD,1.568,ecb,reference
``` ```
In Ledger format: The default output format is CSV, which is suitable for use in spreadsheets and
with other tools. For example, on the command line using
[gnuplot](http://www.gnuplot.info/) we can chart Bitcoin prices:
```bash ```bash
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o ledger | head pricehist fetch coindesk BTC/USD -s 2021-01-01 | \
cut -d, -f1,4 | \
sed 1d | \
gnuplot -p -e '
set datafile separator ",";
set xdata time;
set timefmt "%Y-%m-%d";
set format x "%b\n%Y";
plot "/dev/stdin" using 1:2 with lines title "BTC/USD"
'
```
![BTC/USD prices](example-gnuplot.png)
### Choosing and customizing the output format
You can choose a different output format: `ledger`, `beancount` or
`gnucash-sql`.
```bash
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger
``` ```
``` ```
P 2021-01-04 00:00:00 EUR 1.5928 AUD P 2021-01-04 00:00:00 EUR 1.5928 AUD
@ -123,14 +147,28 @@ P 2021-01-05 00:00:00 EUR 1.5927 AUD
P 2021-01-06 00:00:00 EUR 1.5824 AUD P 2021-01-06 00:00:00 EUR 1.5824 AUD
P 2021-01-07 00:00:00 EUR 1.5836 AUD P 2021-01-07 00:00:00 EUR 1.5836 AUD
P 2021-01-08 00:00:00 EUR 1.5758 AUD P 2021-01-08 00:00:00 EUR 1.5758 AUD
P 2021-01-11 00:00:00 EUR 1.5783 AUD
P 2021-01-12 00:00:00 EUR 1.5742 AUD
P 2021-01-13 00:00:00 EUR 1.5734 AUD
P 2021-01-14 00:00:00 EUR 1.5642 AUD
P 2021-01-15 00:00:00 EUR 1.568 AUD
``` ```
Generate SQL for a GnuCash database and apply it immediately: The formatting options let you control certain details of the output. The
following example removes the time of day (to make it suitable for hledger) and
makes a few other adjustments.
```bash
pricehist fetch ecb EUR/AUD -s 2021-01-04 -e 2021-01-08 -o ledger \
--fmt-time '' --fmt-datesep / --fmt-base € --fmt-quote $ --fmt-symbol left
```
```
P 2021/01/04 € $1.5928
P 2021/01/05 € $1.5927
P 2021/01/06 € $1.5824
P 2021/01/07 € $1.5836
P 2021/01/08 € $1.5758
```
### GnuCash SQL output
You can generate SQL for a GnuCash database and apply it immediately with one
of the following commands:
```bash ```bash
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | sqlite3 Accounts.gnucash pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | sqlite3 Accounts.gnucash
@ -138,12 +176,197 @@ pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | mysql -u username -p
pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | psql -U username -d databasename -v ON_ERROR_STOP=1 pricehist fetch ecb EUR/AUD -s 2021-01-01 -o gnucash-sql | psql -U username -d databasename -v ON_ERROR_STOP=1
``` ```
## Design choices Beware that the GnuCash project itself does not support integration at the
database level, so there is a risk that the SQL generated by `pricehist` will
be ineffective or even damaging for some new version of GnuCash.
To keep things simple, at least for now, `pricehist` provides only univariate In practice, this strategy has been used successfully by other projects.
time series of daily historical prices. It doesn't provide other types of Reviewing the SQL and keeping regular database backups is recommended.
market, financial or economic data, real-time prices, or other temporal
resolutions. Multiple or multivariate series require multiple invocations. ### Source information
Some basic information about each source is available:
```bash
pricehist source ecb
```
```
ID : ecb
Name : European Central Bank
Description : European Central Bank Euro foreign exchange reference rates
URL : https://www.ecb.europa.eu/stats/exchange/eurofxref/html/index.en.html
Start : 1999-01-04
Types : reference
```
Symbol information can be listed for most sources, either as full pairs or as
separate base and quote symbols that work in particular combinations:
```bash
pricehist source ecb --symbols
```
```
EUR/AUD Euro against Australian Dollar
EUR/BGN Euro against Bulgarian Lev
EUR/BRL Euro against Brazilian Real
EUR/CAD Euro against Canadian Dollar
EUR/CHF Euro against Swiss Franc
...
```
It may also be possible to search for symbols:
```bash
pricehist source alphavantage --search Tesla
```
```
TL0.DEX Tesla, Equity, XETRA, EUR
TL0.FRK Tesla, Equity, Frankfurt, EUR
TSLA34.SAO Tesla, Equity, Brazil/Sao Paolo, BRL
TSLA Tesla Inc, Equity, United States, USD
TXLZF Tesla Exploration Ltd, Equity, United States, USD
```
### Inspecting source interactions
You can see extra information, including `curl` commands to reproduce each
request to a source, by adding the verbose option (`--verbose` or `-vvv`):
```bash
pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv
```
```
DEBUG Began pricehist run at 2021-08-12 14:38:26.630357.
DEBUG Starting new HTTPS connection (1): api.coindesk.com:443
DEBUG https://api.coindesk.com:443 "GET /v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05 HTTP/1.1" 200 319
DEBUG curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.25.1' --compressed 'https://api.coindesk.com/v1/bpi/historical/close.json?currency=USD&start=2021-01-01&end=2021-01-05'
DEBUG Available data covers the interval [2021-01-01--2021-01-05], as requested.
date,base,quote,amount,source,type
2021-01-01,BTC,USD,29391.775,coindesk,close
2021-01-02,BTC,USD,32198.48,coindesk,close
2021-01-03,BTC,USD,33033.62,coindesk,close
2021-01-04,BTC,USD,32017.565,coindesk,close
2021-01-05,BTC,USD,34035.0067,coindesk,close
DEBUG Ended pricehist run at 2021-08-12 14:38:26.709428.
```
You can run a logged `curl` command to see exactly what data is returned by the
source:
```bash
pricehist fetch coindesk BTC/USD -s 2021-01-01 -e 2021-01-05 -vvv 2>&1 \
| grep 'DEBUG curl' | sed 's/^DEBUG //' | bash | jq .
```
```json
{
"bpi": {
"2021-01-01": 29391.775,
"2021-01-02": 32198.48,
"2021-01-03": 33033.62,
"2021-01-04": 32017.565,
"2021-01-05": 34035.0067
},
"disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index. BPI value data returned as USD.",
"time": {
"updated": "Jan 6, 2021 00:03:00 UTC",
"updatedISO": "2021-01-06T00:03:00+00:00"
}
}
```
### Use as a library
While not yet optimized for use as a library, you may find `pricehist`'s source
classes useful in your own scripts:
```python
$ python
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pricehist.series import Series
>>> from pricehist.sources.ecb import ECB
>>> series = ECB().fetch(Series("EUR", "AUD", "reference", "2021-01-04", "2021-01-08"))
>>> series.prices
[Price(date='2021-01-04', amount=Decimal('1.5928')), Price(date='2021-01-05', amount=Decimal('1.5927')), Price(date='2021-01-06', amount=Decimal('1.5824')), Price(date='2021-01-07', amount=Decimal('1.5836')), Price(date='2021-01-08', amount=Decimal('1.5758'))]
```
A subclass of `pricehist.exceptions.SourceError` will be raised for any error.
## Terminology
A **source** is the upstream service that can provide a series of prices.
Each **series** of prices is for one pair and price type.
The [**pair**](https://en.wikipedia.org/wiki/Currency_pair) is made up of a
base and a quote, each given as a symbol. Sometimes you will give the base
only, and the quote will be determined with information from the source. The
available pairs, the symbols used in them and the available price types all
depend on the particular source used.
The **base** is the currency or commodity being valued. Each price expresses
the value of one unit of the base.
The **quote** is the unit used to express the value of the base.
A **symbol** is a code or abbreviation for a currency or commodity.
The **prices** in a series each have a date and an amount.
The **amount** is the number of units of the quote that are equal to one unit
of the base.
Consider the following command:
```bash
pricehist fetch coindesk BTC/USD --type close
```
- **`coindesk`** is the ID of the CoinDesk Bitcoin Price Index source.
- **`BTC`** is the symbol for Bitcoin.
- **`USD`** is the symbol for the United States Dollar.
- **`BTC/USD`** is the pair Bitcoin against United States Dollar.
- **`close`** is the price type for last price of each day.
A BTC/USD price of the amount 29,391.775 can be written as
"BTC/USD = 29391.775" or "BTC 29391.775 USD", and means that one Bitcoin is
worth 29,391.775 United States Dollars.
## Initial design choices
To keep things simple, `pricehist` provides only univariate time series of
daily historical prices. It doesn't provide other types of market, financial or
economic data, real-time prices, or other temporal resolutions. Multiple or
multivariate series require multiple invocations.
## Future potential features
While the following features are relatively low value or beyond `pricehist`'s
primary use case, they could be built upon the foundation it has established.
- **Time of day**: Sources sometimes provide specific times for each day's
high/low prices and these could be preserved for output. This would require
changes to how dates are handled internally, clarification of time zone
handling and extension of the time formatting option.
- **Alternate resolutions**: Some sources can provide higher or lower
resolution data, such as hourly or weekly. These could be supported where
available. For other cases an option could be provided for downsampling data
before output.
- **Real-time prices**: These generally come from different soruce endpoints
than the historical data. Real-time prices will usually have a different
price type, such as `last`, `bid` or `ask`. Support for real-time prices
would allow adding sources that don't provide historical data. Start and end
times are irrelevant when requesting real-time prices. A "follow" option
could continuously poll for new prices.
- **Related non-price data**: Trading volume, spreads, split and dividend
events and other related data could be supported. The base/quote/type model
used for prices would work for some of this. Other things may require
extending the model.
- **Multivariate series**: Would allow, for example, fetching
high/low/open/close prices in a single invocation.
- **`format` command**: A command for rewriting existing CSV data into one of
the other output formats.
## Alternatives ## Alternatives

BIN
example-gnuplot.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 32 KiB