Behaviour when no ESP device connected using esptool as a module (ESPTOOL-980)

[mbastida123]

Operating System

Debian GNU/Linux 12 (bookworm)

Esptool Version

esptool==4.8.1

Python Version

Python 3.11.2

Full Esptool Command Line that Was Run

No response

Esptool Output

I'm making a script on python to program some ESP devices via a GUI.

The line:

    esp= esptool.get_default_connected_device(serial_list=['/dev/ttyUSB1'], 
                                                                port=None, 
                                                                connect_attempts=1, 
                                                                initial_baud=921600)

Correctly returns an instance if the ESP device is connected.

However, if there is no ESP device attached to the ESP-PROG the function timeouts and raises FatalError.

This is totally expected behaviour.

The problem is that when this happens the serial port is not correctly closed and so if get_default_connected_device() is placed inside a loop the following calls to said function will never work. Even if you connect an ESP device.
/dev/ttyUSB1 failed to connect: Could not open /dev/ttyUSB1, the port is busy or doesn't exist.

I have tried numerous ways to manually close the port. But the serial instance is not available to me.

Is this an issue or am I doing something wrong?

[Dzarda7]

Hi @mbastida123, the esptool as python package can be used with context manager. We recently added the section into the documentation, so you can take a look here. It should ensure that port is properly closed afterwards. Hope it helps.

[dobairoland]

I'm closing this for inactivity

[mbastida123]

Hi, @dobairoland didn't respond until today because I was on holiday :)

@Dzarda7 thanks for the resource. However, it still doesn't work for me.
To reproduce the error simply add a while True: just before detect_chip and do not connect any ESP device to the ESP-PROG.

The first time the script will correctly timeout on "Connecting......" trying to find a device. However, on the next loop it will be unable to connect to the ESP-PROG via serial. I suspect because the port is left open.

[dobairoland]

Can you please share a complete example we can try? We will reopen this if we can reproduce an issue.

[mbastida123]

Here I go:

test.zip

I have attached the requirements.txt file. Execute programmer.py with a ESP-PROG connected (you may have to tweak the serial port) but no ESP device connected to it.

This is what you should see:

(.venv) pi@rpimarti:~/rpi-programmer/app $ /home/pi/rpi-programmer/app/.venv/bin/python /home/pi/rpi-programmer/app/programmer.py
Binary flashing list: [[0, <_io.BufferedReader name='./binaries/0x0_bootloader.bin'>]]
Connecting......................................
ESP or prog not detectedFailed to connect to Espressif device: No serial data received.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html
[DEBUG] 'None' sent data 'signal => programmer.device_not_detected_abort, type => 1'
Binary flashing list: [[0, <_io.BufferedReader name='./binaries/0x0_bootloader.bin'>]]
ESP or prog not detectedCould not open /dev/ttyUSB1, the port is busy or doesn't exist.
([Errno 11] Could not exclusively lock port /dev/ttyUSB1: [Errno 11] Resource temporarily unavailable)

[DEBUG] 'None' sent data 'signal => programmer.device_not_detected_abort, type => 1'
Binary flashing list: [[0, <_io.BufferedReader name='./binaries/0x0_bootloader.bin'>]]
ESP or prog not detectedCould not open /dev/ttyUSB1, the port is busy or doesn't exist.
([Errno 11] Could not exclusively lock port /dev/ttyUSB1: [Errno 11] Resource temporarily unavailable)

[Dzarda7]

Hi @mbastida123, this is intentional. esptool raises exception if it cannot connected to the device. Your port is still present as you use ESP-PROG, but no device is connected, so the esptool opens the port and tries to connect. If this is not successful, it raises the exception with error. You need to handle the exception by yourself and decide what to do with it. The exception you get is following if I am not mistaken:

esptool.util.FatalError: Failed to connect to Espressif device: No serial data received.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html

The simplest solution for that behavior is this:

from esptool.cmds import detect_chip
import time

# The port of the connected ESP
PORT = "/dev/ttyACM0"
# The binary file
BIN_FILE = "./firmware.bin"
# Flash offset to flash the binary to
FLASH_ADDRESS = 0x10000

def progress_callback(percent):
    print(f"Wrote: {int(percent)}%")

while True:
    try:
        print("Detecting ESP...")
        with detect_chip(PORT) as esp:
            description = esp.get_chip_description()
            features = esp.get_chip_features()
            print(f"Detected ESP on port {PORT}: {description}")
            print(f"Features: {", ".join(features)}")

            esp = esp.run_stub()
            with open(BIN_FILE, 'rb') as binary:
                # Load the binary
                binary_data = binary.read()
                total_size = len(binary_data)
                print(f"Binary size: {total_size} bytes")

                # Write binary blocks
                esp.flash_begin(total_size, FLASH_ADDRESS)
                for i in range(0, total_size, esp.FLASH_WRITE_SIZE):
                    block = binary_data[i:i + esp.FLASH_WRITE_SIZE]
                    # Pad the last block
                    block = block + bytes([0xFF]) * (esp.FLASH_WRITE_SIZE - len(block))
                    esp.flash_block(block, i + FLASH_ADDRESS)
                    progress_callback(float(i + len(block)) / total_size * 100)
                esp.flash_finish()

                # Reset the chip out of bootloader mode
                esp.hard_reset()
    except Exception as e:
        print(f"Error: {e}")
        time.sleep(1)

Sorry for inconvenience, esptool was not meant to be used as a module in the first place so some things are not as easy as it should be, but we plan to change it esptool 5.0.

[mbastida123]

Hi @mbastida123, this is intentional. esptool raises exception if it cannot connected to the device. Your port is still present as you use ESP-PROG, but no device is connected, so the esptool opens the port and tries to connect. If this is not successful, it raises the exception with error. You need to handle the exception by yourself and decide what to do with it. The exception you get is following if I am not mistaken:

esptool.util.FatalError: Failed to connect to Espressif device: No serial data received.
For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html

The simplest solution for that behavior is this:

from esptool.cmds import detect_chip
import time

# The port of the connected ESP
PORT = "/dev/ttyACM0"
# The binary file
BIN_FILE = "./firmware.bin"
# Flash offset to flash the binary to
FLASH_ADDRESS = 0x10000

def progress_callback(percent):
    print(f"Wrote: {int(percent)}%")

while True:
    try:
        print("Detecting ESP...")
        with detect_chip(PORT) as esp:
            description = esp.get_chip_description()
            features = esp.get_chip_features()
            print(f"Detected ESP on port {PORT}: {description}")
            print(f"Features: {", ".join(features)}")

            esp = esp.run_stub()
            with open(BIN_FILE, 'rb') as binary:
                # Load the binary
                binary_data = binary.read()
                total_size = len(binary_data)
                print(f"Binary size: {total_size} bytes")

                # Write binary blocks
                esp.flash_begin(total_size, FLASH_ADDRESS)
                for i in range(0, total_size, esp.FLASH_WRITE_SIZE):
                    block = binary_data[i:i + esp.FLASH_WRITE_SIZE]
                    # Pad the last block
                    block = block + bytes([0xFF]) * (esp.FLASH_WRITE_SIZE - len(block))
                    esp.flash_block(block, i + FLASH_ADDRESS)
                    progress_callback(float(i + len(block)) / total_size * 100)
                esp.flash_finish()

                # Reset the chip out of bootloader mode
                esp.hard_reset()
    except Exception as e:
        print(f"Error: {e}")
        time.sleep(1)

Sorry for inconvenience, esptool was not meant to be used as a module in the first place so some things are not as easy as it should be, but we plan to change it esptool 5.0.

I supose you wrote the response before I uploaded my example.

Anyway, you are correct. And I do catch the exception. The problem is that even after catching the exception it is impossible to connect to the ESP-PROG via serial.

See the log output in my previous message. The first time it works as expected, raising an excepction that I capture as esptool.FatalError. But the second time it doesn't

[Dzarda7]

Yes, sorry, I missed that. You are right, this does not work correctly, I will try to look at it as soon as possible.

[Dzarda7]

@mbastida123 the issue was fixed in this commit, you can try it. You just need to use master branch as a release will be done later. Thanks for reporting this.

[mbastida123]

@mbastida123 the issue was fixed in this commit, you can try it. You just need to use master branch as a release will be done later. Thanks for reporting this.

Thanks for the quick fix! It it working.

However, is it possible that the function write_flash has been deleted?

[Dzarda7]

You cannot use it like this unfortunately. You cannot use write_flash directly. You can see in the documentation, that the code present is equivalent of esptool.py -p /dev/ttyACM0 write_flash 0x10000 firmware.bin so you need to call flash_begin, flash_block and flash_finish respectively.

[mbastida123]

@Dzarda7 just a heads up since I used your example from here:

I think you have an error here:
progress_callback(float(i + len(block)) / total_size * 100)

It should be
progress_callback(min(float(i + len(block)), total_size) / total_size * 100)

To prevent the padding of the last block to produce a progress percentage above 100%

You could even replace len(block) by esp.FLASH_WRITE_SIZE which to me is more intuitive. But that's just cosmetic.

[Dzarda7]

Thanks for noticing this. This serves just as a demonstration, so it is not a big issue. If you are willing to, you can open pull request, it can be fixed here.