From 3cd119dda1455558750817e06671050ef1aa8e8c Mon Sep 17 00:00:00 2001 From: olli Date: Wed, 18 Dec 2024 11:35:55 +0100 Subject: [PATCH] fixes/additional infos --- README.md | 252 +++++++++++------- dabo/dabo-bot.conf | 18 +- dabo/functions/run_strategies.sh | 2 +- dabo/functions/webpage.sh | 1 + strategies/example.strategy.sh | 20 ++ .../example_manage_positions.strategy.sh | 19 ++ 6 files changed, 208 insertions(+), 104 deletions(-) diff --git a/README.md b/README.md index dbe6391..74ba724 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ It is still under development and is currently only partially functional. ## Copyright / License -Copyright (c) 2022-2024 Oliver Bohlen (aka olli) +Copyright (c) 2022-2024 Oliver Bohlen (aka olli/egabosh) The software provided here is called dabo (crypto bot) @@ -25,14 +25,14 @@ You should have received a copy of the GNU General Public License along with dab ## Data sources Various data sources such as finance.yahoo.com and crypto exchanges available via ccxt are used. Please check whether this is legal in your region before use. -- query1.finance.yahoo.com (economic data,...) -- api.coinmarketcap.com (crypto data) -- api.bls.gov (CPI, unemployment rate) -- fred.stlouisfed.org (fed funds rate) -- 30rates.com (forecast) -- fapi.binance.com (OpenInterest,...) -- api.alternative.me (Fear and Greed) -- production.dataviz.cnn.io (Fear and Greed CNN) +- https://query1.finance.yahoo.com (economic data,...) +- https://api.coinmarketcap.com (crypto data) +- https://api.bls.gov (CPI, unemployment rate) +- https://fred.stlouisfed.org (fed funds rate) +- https://30rates.com (forecast) +- https://fapi.binance.com (OpenInterest,...) +- https://api.alternative.me (Fear and Greed) +- https://production.dataviz.cnn.io (Fear and Greed CNN) - https://unpkg.com/lightweight-charts/dist/lightweight-charts.standalone.production.js (TradingView Lightweitgt Charts) - ... @@ -41,7 +41,7 @@ Various data sources such as finance.yahoo.com and crypto exchanges available vi - TradingView Lightweitgt Charts (https://www.tradingview.com/lightweight-charts/ | https://github.com/tradingview/lightweight-charts) - bash, python - several default linux programs like bc, wget,... -- gaboshlib (https://gitea.ds9.dedyn.io/olli/gaboshlib) +- gaboshlib (https://github.com/egabosh/gaboshlib) - ... ## Objective @@ -58,7 +58,7 @@ https://memory-alpha.fandom.com/wiki/Dabo I thought this fits quite well to the cryptotrading world and that's why I chose this name ;-) ## Structure -The Bot is splitted in the following parts: +The Bot is splitted in the following parts/containers: - dabo-bot: Basic Bot with buy and sell decisions based on self-definable strategies - dabo-symbols_ticker: Ticker for current symbols and prices @@ -136,47 +136,57 @@ https://www.debian.org/download https://www.raspberrypi.com/software/operating-systems/ ### 2: Run Ansible Playbooks -On a clean Debian 12 system ypu can run my Ansible Playbooks to use the same environment the bot is developed and running. +On a fresh Debian 12 system you can run my Ansible Playbooks to use the same environment the bot is developed and running. Please have a look what exactly the playbooks are doing if you are unsure. +If the web interface (or matrix/turn) should be accessible via the internet, the server must be accessible on port 80/tcp (acme/letsencrypt) and 443/tcp. +For private/home internet access, this may require port forwarding on your router and registration with a dyndns (DDNS) service. The DDNS service should allow subdomains. + +When using my playbooks and the services should be available to the internet, the hostname must correspond to the ddns name. +Here is an example for the dynu-DDNS service: +- DDNS name: myhost.mywire.com +- dabo-bot-name: dabo.myhost.mywire.com +- matrix-name: matrix.myhost.mywire.com + +You also can use IPv6 if your ISP and router supports this and if you have multiple servers/services on Ports (80/443) but only one IPv4 address to the internet. + #### 2.1 Download basic install script ``` -wget https://gitea.ds9.dedyn.io/olli/debian.ansible.basics/raw/branch/main/install.sh +wget https://raw.githubusercontent.com/egabosh/linux-setups/refs/heads/main/debian/install.sh -O install.sh ``` #### 2.2 define Playbooks -- debian.ansible.basics (https://gitea.ds9.dedyn.io/olli/debian.ansible.basics) - Basic Debian configuration -- Optional: debian.ansible.firewall (https://gitea.ds9.dedyn.io/olli/debian.ansible.firewall) - Firewall for the server based on ufw -- Optional: debian.ansible.runchecks (https://gitea.ds9.dedyn.io/olli/debian.ansible.runchecks) - System checks and notification -- Optional: debian.ansible.backup (https://gitea.ds9.dedyn.io/olli/debian.ansible.backup/src/branch/main/backup.yml) - Backup framework -- Optional: debian.ansible.autoupdate (https://gitea.ds9.dedyn.io/olli/debian.ansible.autoupdate) - Automatic System Updates -- debian.ansible.docker (https://gitea.ds9.dedyn.io/olli/debian.ansible.docker/src/branch/main/docker.yml) - Docker Installation -- debian.ansible.traefik.server (https://gitea.ds9.dedyn.io/olli/debian.ansible.traefik.server) - Traefik Reverse Proxy for Web UI and Letsencrypt Certs -- Optional: debian.ansible.turn.server (https://gitea.ds9.dedyn.io/olli/debian.ansible.turn.server) - Turn Server fpr Audio/Video conferences in Matrix -- Optional: debian.ansible.matrix.server (https://gitea.ds9.dedyn.io/olli/debian.ansible.matrix.server) - Notifications with own Martix Server -- dabo: The Bot itself +- debian/basics/basics.yml (https://github.com/egabosh/linux-setups/tree/main/debian/basics) - Basic Debian configuration +- Optional: debian/firewall/firewall.yml (https://github.com/egabosh/linux-setups/tree/main/debian/firewall) - Firewall for the server based on ufw +- Optional: debian/runchecks/runchecks.yml (https://github.com/egabosh/linux-setups/tree/main/debian/runchecks) - System checks and notification +- Optional: debian/backup/backup.yml (https://github.com/egabosh/linux-setups/tree/main/debian/backup) - Backup framework +- Optional: debian/autoupdate/autoupdate.yml (https://github.com/egabosh/linux-setups/tree/main/debian/autoupdate) - Automatic System Updates +- debian/docker/docker.yml (https://github.com/egabosh/linux-setups/tree/main/debian/docker) - Docker Installation +- debian/traefik.server/traefik.yml (https://github.com/egabosh/linux-setups/tree/main/debian/traefik.server) - Traefik Reverse Proxy for Web UI and Letsencrypt Certs +- Optional: debian/turn.server/turn.yml (https://github.com/egabosh/linux-setups/tree/main/debian/turn.server) - Turn Server fpr Audio/Video conferences in Matrix +- Optional: debian/matrix.server/matrix.yml (https://github.com/egabosh/linux-setups/tree/main/debian/matrix.server) - Notifications with own Martix Server +- https://github.com/egabosh/dabo/raw/refs/heads/main/dabo-ansible.yml: The Bot itself for example: ``` -PLAYBOOKS="debian.ansible.basics - debian.ansible.firewall - debian.ansible.runchecks - debian.ansible.backup - debian.ansible.autoupdate - debian.ansible.docker - debian.ansible.traefik.server - debian.ansible.turn.server - debian.ansible.matrix.server - dabo" +PLAYBOOKS="debian/basics/basics.yml + debian/firewall/firewall.yml + debian/runchecks/runchecks.yml + debian/backup/backup.yml + debian/autoupdate/autoupdate.yml + debian/docker/docker.yml + debian/traefik.server/traefik.yml + https://github.com/egabosh/dabo/raw/refs/heads/main/dabo-ansible.yml" export PLAYBOOKS ``` #### 2.3 Install ansible and run Playbooks +If you run this as user and not as root, the script will install sudo and enable the user to execute commands as root via sudo. To do this, the root password is requested (several times). ``` bash install.sh ``` ### Download Not necessary if you use the dabo Playbook ``` -git clone https://gitea.ds9.dedyn.io/olli/dabo.git +git clone https://github.com/egabosh/dabo.git cd dabo ``` @@ -187,6 +197,8 @@ docker -l warn compose --ansi never build --progress=plain --pull --no-cache --f ``` ### 3. Configuration +Not necessary if you use the dabo Playbook + Edit docker-compose.yml or create docker-compose.override.yml to fit yout needs e.g. domain and network settings or basic auth, e.g. for traefik and letsencrypt: ``` echo ' @@ -227,9 +239,9 @@ networks: ' >docker-compose.override.yml ``` +### 4. Optional: Matrix connection - -Optional: If you use matrix/matrix-commander (https://gitea.ds9.dedyn.io/olli/debian.ansible.matrix.server) and want do receive Matrix-Messages from the bot you can create an SSH-Key to allow sending Matrix-Messages e.g.: +Optional: If you use matrix/matrix-commander (https://github.com/egabosh/linux-setups/tree/main/debian/matrix.server) and want do receive Matrix-Messages from the bot you can create an SSH-Key to allow sending Matrix-Messages e.g.: Automatically done by playbooks. ``` mkdir -p home/.ssh @@ -239,7 +251,7 @@ cat home/.ssh/id_ed25519.pub ``` and add Key on your matrix-Server to the authorized_keys of the matrix-User - +### 5. Add Exchange Create Secrets file for your API Key(s) - file: dabo/.CCXT-ID-secrets @@ -262,54 +274,13 @@ chmod 400 dabo/.binance-secrets Create Config Especially set URL, STOCK_EXCHANGE, FEE, CURRENCY,... to fit your needs. +If you want to use the a testnet of an exchnage if available use TESTNET=true which is the default ``` vim dabo-bot.conf ``` Defaults in dabo/dabo-bot.conf -### 4. Prepare/Create a stretegy - -IMPORTANT!!! - -THE DEFAULT STRATEGY MAY NOT FIT YOUR NEEDS OR WORK PROPERLY. SO YOU CAN LOOSE ALL YOUR MONEY!!! USE ON YOUR OWN RISK!!! - -TEST YOUR OWN STRATEGY COMPREHENSIVELY AND OVER A LOGNER PERIOD OF TIME WITH analyze.sh!!! USE ON YOUR OWN RISK!!! - -Strategie files can be put in the "strategies"-directory the defaults - -There is an example for a buy and a sell strategy file (deactivated by "return 0" in the forst line): -``` -ls strategies/buy.example.conf -ls strategies/sell.example.conf -``` -Aditional strategies can be created with Name -``` -strategies/buy..conf -strategies/sell..conf -``` -e.g named "mannover-sulu-1" for buy strategy and "command-kirk-3" for sell strategy -``` -strategies/buy.mannover-sulu-1.conf -strategies/sell.command-kirk-3.conf -``` - -### Optional: Create individual watch-assets.csv -``` -cp dabo/watch-assets.csv watch-assets.csv -``` -Optional: -You can edit this file if you want do generate warnings or track your asstes/trades. -``` -nano watch-assets.csv -``` - -### Set Rights -Set Rights (UID 10000 for non-root-User in running container): -``` -chown -R 10000:10000 dabo data home strategies dabo-bot.conf watch-assets.csv -``` - ### Operational commands Run/Restart: ``` @@ -317,7 +288,7 @@ docker compose down # if an old instance is running docker compose up -d ``` -Check +List and state of containers: ``` docker compose ps ``` @@ -326,27 +297,74 @@ Logs/Output: ``` docker compose logs -f ``` +Logs/Oputput of a specific container for example dabo-bot: +``` +docker compose logs -f dabo-bot +``` -Update: + +### Update Not necessary if you use the playbooks ``` -# Optinal: Remove local data +# Optional: Remove local data git reset --hard HEAD^ # Remove local commits git clean -fd # Remove local uncommited files # Update and restart -git pull https://gitea.ds9.dedyn.io/olli/dabo.git main -f +git pull https://github.com/egabosh/dabo.git main -f docker compose down docker compose up -d ``` ## Strategies +IMPORTANT!!! + +THE EXAMPLE STRATEGIES MAY NOT FIT YOUR NEEDS OR WORK PROPERLY. SO YOU CAN LOOSE ALL YOUR MONEY!!! USE ON YOUR OWN RISK!!! + +TEST YOUR OWN STRATEGY COMPREHENSIVELY AND OVER A LOGNER PERIOD OF TIME BEST RISK-FREE IN A TESTNET!!! USE ON YOUR OWN RISK!!! + +Strategies are needed for the bot to trade. + +Strategies are located in die stretegies subdir. + You can put your own code into the strategies it will be sourced by the bot. +If you want, you can also use other programming languages or binary code as a strategy and simply start it using your own strategy. -You can use available variables to read (and set) values. +### Example strategies +There are examples for strategy files (deactivated by "return 0" in the beginning): +``` +cat strategies/example.strategy.sh +cat strategies/example_manage_positions.strategy.sh +``` +"example.strategy.sh" creates a score by defined tests and opens a long or short order with stoploss and takeprofit if the score has a defined value. -### Variables with current market values +"example_manage_positions.strategy.sh" watches open positions and switches the stoploss into profit if position is in profit to secure it. + +You can use them and change them to fit your needs. To avoid resets on updates copy them to your own strategies for example: + +``` +cp -p strategies/example.strategy.sh strategies/my-own-trading.strategy.sh +cp -p strategies/example_manage_positions.strategy.sh strategies/my-own-position.strategy.sh +``` + +### Own strategies +Aditional strategies can be created with naming convention *.strategy.sh +``` +strategies/my-new.strategy.sh +strategies/yet-another-new.strategy.sh +``` + +Strategy files should have specific rights - must be readable by the bot: +``` +chown -R 10000:10000 strategies +chmod -R 640 strategies +``` + +### Variables during runtime of the strategies + +You can use available variables/arrays during runtime to read (and set) values. +This arrays are available to runtime in the strategies #### Large associative arrays v and vr (reverse) @@ -364,11 +382,12 @@ ${v[ECONOMY_SP500_rsi14_0]} ${v[ETHUSDT_1w_ema200_0]} ${v[ETHUSDT_1w_macd_histogram_signal_1]} ${v[SOLUSDT_levels_1d]} -$v[$[SOLUSDT_levels_1d_next_up]} -$v[$[ETHUSDT_levels_1w_next_down]} +${v[$[SOLUSDT_levels_1d_next_up]} +${v[$[ETHUSDT_levels_1w_next_down]} ``` -You can find a complete list of available values in the file `data/botdata/values` whic is creates in the runtime of the bot. -An example you can find in example-values. +You can find a complete list of available values in the file `data/botdata/values` which is created in the runtime of the bot. +An long example list you can find in example-values. https://github.com/egabosh/dabo/blob/main/example-values + #### Current price from exchange ${f_tickers_array[SYMBOL]} @@ -377,14 +396,60 @@ ${f_tickers_array[SOLUSDT]} ${f_tickers_array[ETH${CURRENCY}]} ``` +#### Open Orders + +``` +${o[ETHUSDT_present]}=sl_close_long tp_close_long +${o[ETHUSDT_sl_close_long_amount]}=0 +${o[ETHUSDT_sl_close_long_entry_price]}=null +${o[ETHUSDT_sl_close_long_id]}=36b2f404-0b20-4806-bdcc-0dad0daf57e9 +${o[ETHUSDT_sl_close_long_side]}=sell +${o[ETHUSDT_sl_close_long_stoplossprice]}=0 +${o[ETHUSDT_sl_close_long_stopprice]}=3305.98 +${o[ETHUSDT_sl_close_long_takeprofitprice]}=0 +${o[ETHUSDT_sl_close_long_type]}=Stop +${o[ETHUSDT_tp_close_long_amount]}=0 +${o[ETHUSDT_tp_close_long_entry_price]}=null +${o[ETHUSDT_tp_close_long_id]}=850dd169-7387-4be3-ac68-bfce73cfa47d +${o[ETHUSDT_tp_close_long_side]}=sell +${o[ETHUSDT_tp_close_long_stoplossprice]}=0 +${o[ETHUSDT_tp_close_long_stopprice]}=3710.04 +${o[ETHUSDT_tp_close_long_takeprofitprice]}=0 +${o[ETHUSDT_tp_close_long_type]}=MarketIfTouched +``` + +You can find a complete list of available values in the file `data/botdata/values-orders` which is created in the runtime of the bot. + + +#### Open Positions + +``` +${p[ETHUSDT_currency_amount]}=9509.25 +${p[ETHUSDT_current_price]}=3573.79 +${p[ETHUSDT_entry_price]}=3673.31 +${p[ETHUSDT_leverage]}=5 +${p[ETHUSDT_liquidation_price]}=2953.43 +${p[ETHUSDT_pnl]}=-1288.50 +${p[ETHUSDT_pnl_percentage]}=-13.55 +${p[ETHUSDT_side]}=long +${p[ETHUSDT_stoploss_price]}=3305.98 +${p[ETHUSDT_takeprofit_price]}=3710.04 +``` + +You can find a complete list of available values in the file `data/botdata/values-positions` which is created in the runtime of the bot. + + ## Support/Community -New Telegram group for the dabo comunity. +New Telegram group for the dabo community. https://t.me/dabobotcrypto -## Future ideas/featrues and todosa -- Partial StopLoss and TakeProfit -- Fibonacci Levels +## Future ideas/featrues and todos +- PSR Indicator https://de.tradingview.com/script/w4U2xUN7-Pivot-Support-Resistance/ +- public order data if there is such a thing. At what price do people want to enter (limit order)? Where do they want to exit (TakePofit/StopLoss)? Maybe free Binance data helps (https://developers.binance.com/docs/derivatives/usds-margined-futures/market-data/rest-api/Order-Book)? https://www.coinglass.com/de/mergev2/BTC-USDT +- AI implementation. What data can be prepared for training? +- example Grid trading strategy (user specific) +- Fibonacci Levels (which timeframe(s)?) - Forecasts - Chart improvements - Volumeindicator and for example RSI on volume values @@ -395,3 +460,4 @@ https://t.me/dabobotcrypto - Analysis tool for collected historical values to try out buy or sell conditions based on them - Consideration of trading and funding fees - Liquidation Heatmap (https://www.coinglass.com/pro/futures/LiquidationHeatMap) + diff --git a/dabo/dabo-bot.conf b/dabo/dabo-bot.conf index 4d09b62..7f23119 100755 --- a/dabo/dabo-bot.conf +++ b/dabo/dabo-bot.conf @@ -2,16 +2,18 @@ # Webpage URL URL="mydabobot.mydomain" +URL="dabo.$(hostname)" # The exchange we use for using the correct API (CCXI-ID see: https://github.com/ccxt/ccxt) +# for Phemex it is "phemex" for example STOCK_EXCHANGE="NONE" -# Interval in seconds - Should not be lower then 300 +# Interval in seconds - Should not be lower then 300 seconds (5m is the lowest timeframe INTERVAL="300" ## Currency used for trading CURRENCY="USDT" -TRANSFER_CURRENCY="USD" +TRANSFER_CURRENCY="EUR" # symbols that should be traded SYMBOLS="ETH SOL" @@ -19,16 +21,12 @@ SYMBOLS="ETH SOL" ## Signal Group for Notifications NOTIFICATION_GROUP="Dabo-Bot" -## Percent from balance per invest. -# Overwritten by MIN_NOTIONAL+X% from stock if lower -INVEST="5" - -# Stop all trading and sell everything if the complete balance shrinks under this value in ${CURRENCY} -EMERGENCY_STOP="1000" - # Leverage -LEVERAGE="" +LEVERAGE="2" # margin mode (isolated or cross) MARGIN_MODE="isolated" +# testnet of exchange if available (true or false). false for trades with real money +TESTNET=true + diff --git a/dabo/functions/run_strategies.sh b/dabo/functions/run_strategies.sh index bef140b..0ae01f8 100644 --- a/dabo/functions/run_strategies.sh +++ b/dabo/functions/run_strategies.sh @@ -30,7 +30,7 @@ function run_strategies { get_symbols_ticker get_values ${f_symbols_array_trade[*]} - for f_strategy in $(find /dabo/strategies -type f -name "*strategy*" ! -name "\.*") + for f_strategy in $(find /dabo/strategies -type f -name "*.strategy.sh" ! -name "\.*") do if ! bash -n "${f_strategy}" >$g_tmp/strat_bash_error 2>&1 then diff --git a/dabo/functions/webpage.sh b/dabo/functions/webpage.sh index ae2e091..f5bdb69 100644 --- a/dabo/functions/webpage.sh +++ b/dabo/functions/webpage.sh @@ -24,6 +24,7 @@ function webpage { webpage_transactions get_symbols_ticker + charts # create status webpage echo " diff --git a/strategies/example.strategy.sh b/strategies/example.strategy.sh index e3365f4..c28f0c4 100644 --- a/strategies/example.strategy.sh +++ b/strategies/example.strategy.sh @@ -1,3 +1,23 @@ +#!/bin/bash + +# Copyright (c) 2022-2024 olli +# +# This file is part of dabo (crypto bot). +# +# dabo is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# dabo is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with dabo. If not, see . + + # Example strategy g_echo_note "EXAMPLE Strategy" diff --git a/strategies/example_manage_positions.strategy.sh b/strategies/example_manage_positions.strategy.sh index 6d89bdd..c104eea 100644 --- a/strategies/example_manage_positions.strategy.sh +++ b/strategies/example_manage_positions.strategy.sh @@ -1,3 +1,22 @@ +#!/bin/bash + +# Copyright (c) 2022-2024 olli +# +# This file is part of dabo (crypto bot). +# +# dabo is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# dabo is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with dabo. If not, see . + # Example strategy for managing open positions g_echo_note "EXAMPLE Strategy for managing open positions"