CCXTFeed _fetch_ohlcv() infinite loop

[quanatee]

The line allows the data collection to move forward, otherwise it causes an infinite loop after the first cycle of collection.

It is actually a legacy line from Ed's work.

Here is the differences for me via print(self.symbol, dlen, len(self._data)) in the while loop of _fetch_ohlcv()

Behavior without line:

ETH/BTC 0 999
ETH/BTC 999 999
ETH/BTC 998 1867
ETH/BTC 1867 1867
ETH/BTC 1866 1866
ETH/BTC 1865 1865
ETH/BTC 1864 1864
ETH/BTC 1863 1863

self._data length does not increase when fetching ohlcv, causing an infinite loop

Behavior with line:

ETH/BTC 0 999
ETH/BTC 999 1998
ETH/BTC 1998 2997
ETH/BTC 2997 3996
ETH/BTC 3996 4670
ETH/BTC 4670 4670

`self._data` increase, eventually triggering strategy to start.

[Dave-Vallance]

@rymdluo

Thanks for the pull request! I will review in the coming weeks and attempt to merge it or get back to you with questions.

Appreciate the contribution!

[Dave-Vallance]

@rymdluo

I have tested this for a while whilst working on other things and not noticed any adverse affects. However, I think I am missing the point - Sorry!

Where in the while loop are you putting that print statement? My output looks quite different to yours.

Also, what exchange are you testing on?

At the moment, I am seeing the data length increase on Kraken with or without the edit.

[quanatee]

I think I've placed it just above the if dlen == len(self._data): break, where it checks if it has filled enough data.

I've only tested it in Binance.

[Dave-Vallance]

@rymdluo Thanks! I will take a look on Binance.

[cryptorevizor]

Hello, it really works with binance. I spend 2 days for solving this problem and debugging. Thanks a lot @rymdluo

[cryptorevizor]

            if tstamp > self._last_ts:
                if self.p.debug:
                    print('Adding: {}'.format(ohlcv))
                self._data.append(ohlcv)
                self._last_ts = tstamp
            since = tstamp + 1

        if dlen == len(self._data):
            break

[grantcause]

I have this same error. I have tried to fix this for 2 days now.
I have tried to add the line above as indicated but having made the change I cannot get the ccxtbt module to reload and recognise the change. I have breakpoints in ccxtfeed set that never get hit.

To reload the module I am using:

import ccxtbt
import importlib
importlib.reload(ccxtbt)
from ccxtbt import *

My program never leaves the fetch_ohlc loop in ccxtfeed.py
Fetching: BTC/USDT, TF: 5m, Since: 1601769300001, Limit: 50
2020-10-22 08:34:32.012589 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601784000001, Limit: 50
2020-10-22 08:34:42.288041 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601798700001, Limit: 50
2020-10-22 08:34:52.549484 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601813400001, Limit: 50
2020-10-22 08:35:02.821857 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601828100001, Limit: 50
2020-10-22 08:35:13.129360 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601842800001, Limit: 50
2020-10-22 08:35:23.402750 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601857500001, Limit: 50
2020-10-22 08:35:33.673100 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601872200001, Limit: 50
2020-10-22 08:35:43.942537 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601886900001, Limit: 50
2020-10-22 08:35:54.222886 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601901600001, Limit: 50
2020-10-22 08:36:04.498007 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601916300001, Limit: 50
2020-10-22 08:36:14.765483 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601931000001, Limit: 50
2020-10-22 08:36:25.028083 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601945700001, Limit: 50
2020-10-22 08:36:35.314126 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601960400001, Limit: 50
2020-10-22 08:36:45.587475 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601975100001, Limit: 50
2020-10-22 08:36:55.862849 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1601989800001, Limit: 50
2020-10-22 08:37:06.141953 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602004500001, Limit: 50
2020-10-22 08:37:16.427977 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602019200001, Limit: 50
2020-10-22 08:37:26.696429 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602033900001, Limit: 50
2020-10-22 08:37:36.961982 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602048600001, Limit: 50
2020-10-22 08:37:47.229439 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602063300001, Limit: 50
2020-10-22 08:37:57.505693 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602078000001, Limit: 50
2020-10-22 08:38:07.807353 - fetch_ohlcv - Attempt 0
Fetching: BTC/USDT, TF: 5m, Since: 1602092700001, Limit: 50

Could you please merge this change with the ccxtbt master repository so I can try reinstalling the ccxtbt module?

Thank you for your assistance it is greatly appreciated.