olli/dabo

olli 244667d11a yahoo disables free history api - using coinmarketcap api now

2024-09-18 21:23:40 +02:00

14 KiB

Raw Permalink Blame History

Dabo (crypto bot)

The software provided here is called dabo (crypto bot).

Warning / Disclaimer

The software provided here does not guarantee any profits or function as well as sufficient security. Use at your own risk!!! This is a private project, which is based on amateur knowledge. Trading cryptocurrencies involves an enormous amount of risks and is considered highly speculative. It is strongly recommended to deal intensively with the subject and this bot before using it. Also, when using the possibility should be considered that due to an unfavorable market development, technical errors, bugs or other reasons, the entire invested capital can be lost. This software should therefore only be used if it is justifiable to lose the entire invested capital!

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

It is still under development and is currently only partially functional.

Copyright / License

The software provided here is called dabo (crypto bot)

dabo is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

dabo is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with dabo (see COPYING file). If not, see http://www.gnu.org/licenses/.

Data sources

Various data sources such as finance.yahoo.com and crypto exchanges available via ccxt are used. Please check whether this is legal in your region before use.

query1.finance.yahoo.com (economic data,...)
api.coinmarketcap.com (crypto data)
30rates.com (forecast)
fapi.binance.com (OpenInterest,...)
api.alternative.me (Fear and Greed)
https://unpkg.com/lightweight-charts/dist/lightweight-charts.standalone.production.js (TradingView Lightweitgt Charts)
...

dependencies to other software

CCXT (https://www.ccxt.com | https://github.com/ccxt/ccxt)
TradingView Lightweitgt Charts (https://www.tradingview.com/lightweight-charts/ | https://github.com/tradingview/lightweight-charts)
bash, python
several default linux programs like bc, wget,...
gaboshlib (https://gitea.ds9.dedyn.io/olli/gaboshlib)
...

Objective

The dabo-bot is intended to help make and execute timely buy and sell decisions automatically in the fast-paced crypto environment.

These decisions are made using one or more self-definable strategies. Various market data are available as a basis for decision-making, such as price trends, RSI, MACD and EMA indicators of various time intervals, Fear and Greed Index, S&P500 data,...

Naming

The name Dabo comes from the Star Trek universe. Dabo was a roulette-style game of chance developed by the Ferengi. More information here: https://memory-alpha.fandom.com/wiki/Dabo I thought this fits quite well to the cryptotrading world and that's why I chose this name ;-)

Structure

The Bot is splitted in the following parts:

dabo-bot: Basic Bot with buy and sell decisions based on self-definable strategies
dabo-symbols_ticker: Ticker for current symbols and prices
dabo-ohlcv-candles: Continuously collect OHLCV data
dabo-indicators: Continuously calculate indicators and other data
dabo-orders: Continuously collect current orders placed by bot and others
dabo-transaction-history: Continuously creates overview of historic trades
dabo-webpage: Continuously creates webpage overview (readonly)
dabo-web: Webserver for webpage overview

Each part runs parallel to the others in its own docker-container.

Features

General:

theoretically compatible to supported CCXT exchanges. Tested with Phemex Contract trading. Please let me know your experiences with other exchanges.
parallel processing in separate non-root docker containers
Notifications via Matrix Messenger to groups

Dabo Bot

place limit and market orders and supports ...
... leverage trading
... short trades
... margins cross and isolated
... stoploss and takeprofit
multiple different strategies possible at the same time

Dabo Symbol Ticker

automatically finds all supported tokens and pairs and fetches continiously ...
... symbols
... and prices

Dabo OHLCV Candle data

OHLCV = Open, High, Low, Close and Volume of a time unit

time units 1w, 1d, 4h, 1h, 15m and 5m
4h, 1h, 15m and 5m from the respective stock exchange
1d and 1w data from coinmarketcap to have longer terms
economic data from yahoo finance

Dabo Indicators

data per time unit
time units 1w, 1d, 4h, 1h, 15m and 5m
self-calculated EMA12, 26 ,50, 100, 200, 400 and 800
self-calculated RSI5, 14, 21
self-calculated MACD
self-calculated significant levels (support/resist)

Dabo Market Data

Yahoo Finance
CoinMarketCyp

Dabo Orders

Dabo Transaction History

Support of additional Exchnages/Brokers JustTrade (CSV-Import) and Bitpanda (API+CSV-Import)
German tax calculation

Dabo Webpage

ReadOnly Overview

Why bash?

Yes, bash is not a language in which you write something like this in a normal way.

To be honest, I'm more of a Linux admin than a programmer, so I do a lot of my work using bash. Simply it's the language I'm most familiar with.

Originally this project was supposed to be a simple script to monitor prices of equities and ETFs and set alarms at certain marks. Over time it has expanded a lot and a complete rewrite in another language would have meant a lot of additional work. So it has stayed that way until now.

Finally, it's a hobby project and I have to see how and when I can find time for it, because there also has to be time for family, friends, work and other hobbies. If there is someone who would like to rewrite this bot in e.g. Python, I would be happy to support them as best I can with this task. Just let me know.

How to use/install

Linux knowledge required!

Needed a running Docker install. Traefik suggested, see

https://gitea.ds9.dedyn.io/olli/debian.ansible.docker

https://gitea.ds9.dedyn.io/olli/debian.ansible.traefik.server

Download

git clone https://gitea.ds9.dedyn.io/olli/dabo.git
cd dabo

Build container

docker -l warn compose --ansi never build --progress=plain --pull --no-cache --force-rm

Configure

Edit docker-compose.yml or create docker-compose.override.yml to fit yout needs e.g. domain and network settings or basic auth, e.g. for traefik and letsencrypt:

echo '
services:

  dabo-bot:
    networks:
      - YOURNETWORK

  dabo-web:
    labels:
      - traefik.enable=true
      # DOMAIN
      - traefik.http.routers.dabo-YOURINSTANCENAME.rule=Host(`YOURDOMAIN`)
      - traefik.http.routers.dabo-YOURINSTANCENAME.entrypoints=https
      - traefik.http.routers.dabo-YOURINSTANCENAME.tls=true
      # Proxy to service-port
      - traefik.http.services.dabo-YOURINSTANCENAME.loadbalancer.server.port=80
      - traefik.http.routers.dabo-YOURINSTANCENAME.service=dabo-YOURINSTANCENAME
      # cert via letsencrypt
      - traefik.http.routers.dabo-YOURINSTANCENAME.tls.certresolver=letsencrypt
      # activate secHeaders@file & basic auth
      - traefik.http.routers.dabo-YOURINSTANCENAME.middlewares=secHeaders@file,dabo-YOURINSTANCENAME-basicauth
      # Generate crypted password string with: echo $(htpasswd -nB YOURUSER) | sed -e s/\\$/\\$\\$/g
      - traefik.http.middlewares.dabo-YOURINSTANCENAME-basicauth.basicauth.users=YOURUSER:YOUR-GENERATED-CRYPTED-PASSWORD-STRING
      # Traefik network
      - traefik.docker.network=traefik
    networks:
      - traefik

networks:
  YOURNETWORK:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: YOURBRIDGE
  traefik:
    external: true

' >docker-compose.override.yml

Optional: If you use matrix/matrix-commander (https://gitea.ds9.dedyn.io/olli/debian.ansible.matrix.server) and want do receive Matrix-Messages from the bot you can create an SSH-Key to allow sending Matrix-Messages e.g.:

mkdir -p home/.ssh
ssh-keygen -f home/.ssh/id_ed25519 -N "" -t ed25519
chmod 700 home/.ssh
cat home/.ssh/id_ed25519.pub

and add Key on your matrix-Server to the authorized_keys of the matrix-User

Create Secrets file for your API Key(s)

file: dabo/.CCXT-ID-secrets

CCXI-ID see: https://github.com/ccxt/ccxt

Examples:

# for Phemex 
echo 'local API_SECRET="YOUR_API_SECRET_FROM_PHEMEX"
local API_KEY="YOUR_API_KEY_FROM_BINANCE"
chmod 400 dabo/.phemex-secrets

# for Binance
echo 'local API_SECRET="YOUR_API_SECRET_FROM_BINANCE"
local API_KEY="YOUR_API_KEY_FROM_BINANCE" '>dabo/.binance-secrets
chmod 400 dabo/.binance-secrets

Create Config Especially set URL, STOCK_EXCHANGE, FEE, CURRENCY,... to fit your needs.

vim dabo-bot.conf

Defaults in dabo/dabo-bot.conf

Prepare/Create a stretegy

IMPORTANT!!!

THE DEFAULT STRATEGY MAY NOT FIT YOUR NEEDS OR WORK PROPERLY. SO YOU CAN LOOSE ALL YOUR MONEY!!! USE ON YOUR OWN RISK!!!

TEST YOUR OWN STRATEGY COMPREHENSIVELY AND OVER A LOGNER PERIOD OF TIME WITH analyze.sh!!! USE ON YOUR OWN RISK!!!

Strategie files can be put in the "strategies"-directory the defaults

There is an example for a buy and a sell strategy file (deactivated by "return 0" in the forst line):

ls strategies/buy.example.conf
ls strategies/sell.example.conf

Aditional strategies can be created with Name

strategies/buy.<name>.conf
strategies/sell.<name>.conf

e.g named "mannover-sulu-1" for buy strategy and "command-kirk-3" for sell strategy

strategies/buy.mannover-sulu-1.conf
strategies/sell.command-kirk-3.conf

Create individual watch-assets.csv

cp dabo/watch-assets.csv watch-assets.csv

Optional: You can edit this file if you want do generate warnings or track your asstes/trades.

nano watch-assets.csv

Set Rights

Set Rights (UID 10000 for non-root-User in running container):

chown -R 10000:10000 dabo data home strategies dabo-bot.conf watch-assets.csv

Operational commands

Run/Restart:

docker compose down   # if an old instance is running 
docker compose up -d

Check

docker compose ps

Logs/Output:

docker compose logs -f

Update:

# Optinal: Remove local data
git reset --hard HEAD^   # Remove local commits
git clean -fd            # Remove local uncommited files

# Update and restart
git pull https://gitea.ds9.dedyn.io/olli/dabo.git main -f
docker compose down 
docker compose up -d

dabo-bot.sh is the bot that trades and collects the quotes and analyze.sh is the tool with which you can try out strategies with the historical data. The configuration files are called dabo-bot.conf and analyze.conf. analyze.sh also uses bot.conf but its variables are overwritten by analyze.conf if duplicated.

A Binance or OneTrading.com (ex BitpandaPro) account must exist and the API must be enabled. The access data is stored in the file .binance-secrets in the project directory in the variables API_SECRET and API_KEY. The access rights to this file should be set to the minimum necessary for security reasons.

Strategies

You can put your own code into the strategies it will be sourced by the bot.

You can use available variables to read (and set) values.

Variables with current market values

Large associative arrays v and vr (reverse)

${v[SYMBOL_TIMEFRAME_ITEM_NUMBER]}

SYMBOL: Crypto-Symbol for examle ETHUSDT or ECONOMY_DXY ECONOMY_DOWJONES ECONOMY_SP500 ECONOMY_NASDAQ ECONOMY_MSCIEAFE ECONOMY_10YRTREASURY ECONOMY_OIL ECONOMY_GOLD ECONOMY_MSCIWORLD ECONOMY_OILGAS ECONOMY_KRE ECONOMY_EUR-USD
TIMEFRAME: 5m,15m,1h,4h,1d,1w
ITEM: date,open,high,low,close,volume,change,ath,ema12,ema26,ema50,ema100,ema200,ema400,ema800,rsi5,rsi14,rsi21,macd,macd_ema9_signal,macd_histogram,macd_histogram_signal,macd_histogram_max,macd_histogram_strength
NUMBER: 0=latest; 1=second latest