Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rucio list-dataset-replicas throws HTTP 413 for some datasets #681

Open
haozturk opened this issue Dec 13, 2023 · 1 comment · May be fixed by dmwm/rucio-flux#288
Open

rucio list-dataset-replicas throws HTTP 413 for some datasets #681

haozturk opened this issue Dec 13, 2023 · 1 comment · May be fixed by dmwm/rucio-flux#288
Labels
bug

Comments

@haozturk
Copy link
Contributor

We've been reported that rucio list-dataset-replicas throws HTTP error with status code 413 for a dataset [1]:

$ rucio list-dataset-replicas cms:/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW
2023-12-13 13:52:18,763	ERROR	An unknown exception occurred.
Details: no error information passed (http status code: 413)

I suspect, it's due to the fact it has too many files as status code 413 suggest "Content Too Large". This is a minor issue. We can look into it if we observe it for more datasets. I just wanted to have a record of this.

[1] https://its.cern.ch/jira/browse/CMSPROD-126
@haozturk haozturk added the bug label Dec 13, 2023
@haozturk haozturk mentioned this issue May 23, 2024
Open
@labkode
Copy link
Member

labkode commented May 23, 2024

The request is roughly ~4.5MB (Content-Length: 4526110), see below.

Rucio uses for the server component Apache server, which default body size is unlimited (~ before June 2022) and 1GB afterwards.

Common ingresses are NGINX (default limit 1 MB) and Traefik (unlimited).

So, smells like infrastructure limit. Bumping ingress limits should fix the issue.

sendto(3, "POST /replicas/datasets_bulk HTTP/1.1\r\nHost: cms-rucio.cern.ch\r\nUser-Agent: rucio-clients/32.3.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: Keep-Alive\r\nX-Rucio-Auth-Token: gonzalhu-/DC=ch/DC=cern/OU=Organic
 Units/OU=Users/CN=gonzalhucms/CN=756361/CN=Hugo Gonzalxxx\r\nX-Rucio-VO: def\r\nX-Rucio-Script: rucio::list-dataset-replicas\r\nContent-Length: 4526110\r\n\r\n", 420, 0, NULL, 0) = 420
sendto(3, "{\"dids\": [{\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#000083ba-859a-11e6-a887-001e67abef8c\"}, {\"scope\": \"cms\", \"name\": \"
/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#00009d64-859a-11e6-a887-001e67abef8c\"}, {\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80
X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#0007448e-85c7-11e6-b48c-002590494fb0\"}, {\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RA
W#00349c92-84d0-11e6-a104-002590494c06\"}, {\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#0034c050-84d0-11e6-a104-002590494c06\"}, {\"scope\": \
"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_Tranche"..., 4526110, 0, NULL, 0) = 4526110
munmap(0x7fc452269000, 4530176)         = 0
ioctl(3, FIONBIO, [0])                  = 0
recvfrom(3, "HTTP/1.1 413 Request Entity Too Large\r\nDate: Thu, 23 May 2024 14:23:56 GMT\r\nContent-Type: text/html\r\nContent-Length: 176\r\nConnection: close\r\n\r\n<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n", 8192, 0, NULL, NULL) = 319
close(3)

labkode added a commit to dmwm/rucio-flux that referenced this issue May 23, 2024
@labkode
Increase request body size to 8M for NGINX ingress.
5fe9f44 
Should fix dmwm/CMSRucio#681
@labkode labkode linked a pull request May 23, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Increase request body size to 8M for NGINX ingress. dmwm/rucio-flux
2 participants
@labkode @haozturk