git reset fails sometimes when connecting to gitea forge

[lunedis]

Component

server, agent

Describe the bug

I have been using woodpecker-CI for multiple years now in our company, and we rarely had any issues.

Ever since two days ago, pipelines sometimes fail with an error connecting to the gitea forge:

+ git init --object-format sha1 -b main
Initialized empty Git repository in /woodpecker/src/GITEA_URL/ORGANIZATION/REPOSITORY/.git/
+ git config --global --replace-all safe.directory /woodpecker/src/GITEA_URL/ORGANIZATION/REPOSITORY
+ git remote add origin https://GITEA_URL/ORGANIZATION/REPOSITORY.git
+ git fetch --no-tags --depth=1 --filter=tree:0 origin +caf9334a3d53b3[2](https://CI_URL/repos/73/pipeline/471#L2)f4c382044ca4ff5d5982d4e58:
From https://GITEA_URL/ORGANIZATION/REPOSITORY
 * branch            caf9[3](https://CI_URL/repos/73/pipeline/471#L3)34a3d53b32f4c382044ca4ff5d5982d4e58 -> FETCH_HEAD
+ git reset --hard -q caf9334a3d53b32f4c382044ca4ff5d5982d4e58
fatal: unable to access 'https://GITEA_URL/ORGANIZATION/REPOSITORY.git/': Failed to connect to GITEA_URL port [4](https://CI_URL/repos/73/pipeline/471#L4)43 after 2002 ms: Could not connect to server
fatal: could not fetch 374dd4d87[5](https://CI_URL/repos/73/pipeline/471#L5)f23bb4a2e8b631762d933d12d78d27 from promisor remote
exit status 128

Restarting the pipeline often fixes the issue.

I am failing to find any reasons to why this is happening. It seems to connect to gitea normally before (it loads the pipeline, runs the first fetch, and it works on restart). I have updated to woodpecker-next and the issue stays the same. I have not updated neither woodpecker or gitea in the past few days (just a few weeks ago).

I had a similiar issue half a year ago, when my renovate-bot would open many pull requests simultaneously and it would fail with a "Cannot read username" message.

Is this a problem with woodpecker? Am I running into a gitea rate limit? I am not running fail2ban on my host.

Steps to reproduce

Hard to reproduce sadly :(

Expected behavior

No response

System Info

woodpeckerci/plugin-git: 2.6.0
woodpecker version: next-defe6c418f

Additional context

No response

Validations

• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]

[zc-devs]

This is how it looks on healthy clone:

+ git init --object-format sha1 -b test-branch
Initialized empty Git repository in /woodpecker/src/gitea.domain.tld/wp/test-repo/.git/
+ git config --global --replace-all safe.directory /woodpecker/src/gitea.domain.tld/wp/test-repo
+ git remote add origin https://gitea.domain.tld/wp/test-repo.git
+ git fetch --no-tags --depth=1 --filter=tree:0 origin +c0f44f662bf9ee5e53356a1fc67bc8e0595f5e0a:
From https://gitea.domain.tld/wp/test-repo
 * branch            c0f44f662bf9ee5e53356a1fc67bc8e0595f5e0a -> FETCH_HEAD
+ git reset --hard -q c0f44f662bf9ee5e53356a1fc67bc8e0595f5e0a
+ git submodule update --init --recursive --depth=1 --recommend-shallow
+ git lfs fetch
fetch: Fetching reference refs/heads/test-branch
+ git lfs checkout

Woodpecker next-2e490f3c7b, plugin-git 2.6.0, Gitea 1.22.4.

---

caf9334a3d53b32f4c382044ca4ff5d5982d4e58
caf9 3 34a3d53b32f4c382044ca4ff5d5982d4e58
caf9 [3] (https://CI_URL/repos/73/pipeline/471# L3 )34a3d53b32f4c382044ca4ff5d5982d4e58
caf9[3](https://CI_URL/repos/73/pipeline/471#L3)34a3d53b32f4c382044ca4ff5d5982d4e58

Failed to connect to GITEA_URL port 443
Failed to connect to GITEA_URL port [4] (https://CI_URL/repos/73/pipeline/471# L4 )43
Failed to connect to GITEA_URL port [4](https://CI_URL/repos/73/pipeline/471#L4)43

and so on

[lunedis]

I have changed our gitea instance to use a postgres database instead of the default sqlite, and the problems seems to have gone away. I am however not sure how that changes anything, as the problem was in the fetch, and I think that does not hit the database? I am stumped...

[zc-devs]

fetch does not hit the database

I think so.

What do you have between Server and Agent, as well as between clone container and Gitea?
Could you check environment variables, passed to failed clone container? Pod definition on Kubernetes.
Also, attach debug logs (Server, Agent, Gitea, HTTP proxy), please. Maybe there is setting like debug: true in clone plugin also. Do you have the same logs from container directly as the logs of clone step from Server?
Show your pipeline (or example) and configuration of the components.