-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
better sanitize body of PUT requests #103
Comments
@belforte , it is long battle with DBS codebase to align with proper data-types. The list in JSON ishould be represented as I'll leave it up to @d-ylee and @todor-ivanov to straight this out and properly align data-types in DBS server and client calls. |
I am all for clean and consistent (and documented) interfaces, and have struggled with old DBS python since the tiime it was returning ("%s"%somedictionary) and calling it JSON. But the main problem here is that ["file1","file2"] made the server invalidate every file ! |
@belforte , you should properly define the logic then, e.g. if payload provides list of LFNs (in
In this request I understand the intention of the client and we can work on implementation of what needs to be done. Please translate your request in similar form and use curl and bare HTTP language to provide logic of the request along with payload and what you expect from such request. |
sorry I was not clear. The server was not "invaliating all files in the list which I gave", the server invalidated ALL FILES IN THE DB. Yesterday I had to revalidate files in O(500) user datasets which surely I never indicated in my attempts. This happened around Oct 25 00:30 CET (server local time i.e.) in case someone is willing to look at logs. On the query format, I defer to Dennis, Todor and possibly you to decide what is better from DBS2Go side. I will happily adapt as soon as I am given an example ! |
@belforte , thanks for clarification. I had a look at phys03 DB and here is what I see:
How would you like to deal with this information? May be we can ask CERN DBA to restore a DB backup for this table? If so I suggest you open yet another ticket if any actions are required on DB side. This ticket is only to fix the PUT API logic and settle down on JSON format. |
Thanks @vkuznet As to correcting the situation, DBA's usually avoid editing DB for us , and going back to status of 1 week ago would give a big problem about how to find and recover data which was inserted since. I have asked CMS DBS Oracle support people to isssue and SQL query like this (pseudo code) which will fix:
AFAIU, it should be easy. Well... I asked Yuyi, she said that Oracle side has been passed to Dennis, then Dennis said that he's transitioning all his responsibilities on DBS to Todor. 🤷♂️ In the meanwhile, whenever a user discovers that their favorite dataset has all files invalidated, they can fix via |
if Dennins/Todor like to have a GH issue about that, I will be happy to open. So far only Yuyi replied. |
Hi @belforte , Sorry for not replying immediately. I have not yet gotten access to the Database but will make all my best to do so today. Please do create another issue just for restoring the affected oracle Oracle records and will mark it as operational and will get to it with my highest priority. |
ciao @todor-ivanov . No worry. I had asked for Oracle-side help to Yuyi who CC-ed Dennis (she did not know about your involvement) and Dennis is now looking at fixing records in DB. thanks |
with reference to #102 (comment) and #102 (comment)
I realized that DBS was not hanging, but busy invalidating all files in the DB, see https://cms-talk.web.cern.ch/t/dbs-phys03-files-wrongly-invalidated/31152/1
and references in there.
I re-state the problem here.
At least in case of a PUT call to
/DBSWriter/files
, the body of the put is expected to be in the format"file1,file2,file3,...fileN" i.e. a single string containing a comna-separated list of file names. I was mislead by the terse description in the documentation "a list of file names" and had initially passed a JSON list
["file1","file2","file3",..."fileN"]`Apparently that triggered DBS server to act on all files !
Proper behavior should be to return an error message (better if clear) and ignore the command.
I have no idea if other API's may be similarly affected by bad formattin of the body of the query.
@todor-ivanov @d-ylee
The text was updated successfully, but these errors were encountered: