Repository navigation
Pool migration ignores rdonly status of target #7114
Description
Dear dCache devs,
I have a strong impression that the "migration move" function of a pool ignores the rdonly (read only) status of a target pool.
This is a bit impractical when you have two pools that you want to drain simultaneously with the poolgroup as a target. They keep on writing files to each other, so it's difficult to get them to a stage where they are empty and can be decommissioned.
So, what I do is set the source pools to rdonly, in the PoolManager:
psu set pool shark11_atlasdisk rdonly
psu set pool shark12_atlasdisk rdonly
Then I drain them both like this:
migration move -concurrency=6 -target=pgroup atlas_writediskpools
And then they are writing files to many pools, including each other. I wouldn't expect that, since they are rdonly. It's actually worse: because they are getting emptier, they are becoming increasingly attractive target pools for each other's migrations.
I work around this by setting the max diskspace to 10 (bytes) repeatedly, so that they don't have any available space to write into. But that's a bit clumsy.
Cheers,
Onno
Activity
paulmillar commentedon Apr 12, 2023
I believe you are right: the
migrationcommand does ignore any read-only status of the target pool.One could argue this is a bug (the pool is read-only, so should accept any more data), but also one could argue this isn't a bug (the admin told me to do this, so I'm overriding the pool's settings).
For now, I'm marking this request an enhancement, but somebody should decide whether this behaviour is actually a bug.
If it isn't a bug then one possible solution would be to add an option to the
migrationcommand so that, by default it honours the pools settings but there's an option (something like-f--force) that overrides the pool settings. One could even imagine that option taking an argument (e.g., a list of pools) so that only those pools are overridden, or that the command only overrides the status of destination pools, or ..ageorget commentedon Apr 12, 2023
Hi Onno,
I'm also facing this when I have to take out servers for replacement.
The easier work around I found is to use the migration options
-includeto target only the new installed servers or-excludeto avoid migration between take out servers. And of course set them in readonly mode to avoid new files written by clients.Cheers
Adrien
kofemann commentedon Apr 12, 2023
The main misunderstanding comes from the fact, that
psu set pool shark11_atlasdisk rdonlysets the pool ONLY read only for PoolManager, e.g only for user-initiated requests. The pool itself stays read-write. Thus, all request that doesn't use pool manager and directly talks to pool can write to it. So, if you need pool to operate in read-only mode, then you should do so on the poolpool disable -rdonlykofemann commentedon Apr 12, 2023
I would say:
works as expected. But it looks like that too many admins run into this situation. So maybe we need to update psu to disable the pool unless an extra--softoption is specified.paulmillar commentedon Apr 12, 2023
Good catch. I missed that it was a pool-manager
psu set poolcommand that was disabling the pool.So, I agree with @kofemann .
Just to add my 2c-worth, another option might be to update pools so they have an extra state (or extra states) where the pool would be disabled for client-triggered requests (upload/download/stage), but would still allow sys-admin triggered requests. This would have broadly the same effect as the current
psu set poolcommand, but the pool would be the authoritative source of information (pool-manager could cache this info, but that would be only a cache).That would be a bigger change, but (I think) would bring clarity to a somewhat confusing situation.
onnozweers commentedon Apr 12, 2023
Thanks guys, I wasn't aware that the poolmanager rdonly was something different than the pool rdonly. Thanks for clearing that up.
I like @kofemann's idea to have the poolmanager's
psu set pool shark11_atlasdisk rdonlycommand update also the pool status (unless specified otherwise). For now I guess I can use thepool disable -rdonlyas workaround.onnozweers commentedon Apr 12, 2023
Disabling downloads from a pool while draining it with a
migration movemight be a bit impractical. I prefer the users to be able to access their data while it is waiting to be migrated. In a case where you'd really want the user to back off while draining the pool (e.g. for performance reasons) you can always set the number of movers to a low value or even zero.kofemann commentedon Apr 12, 2023
this issues looks like a good candidate to test github discussions :). So, I here am I. 😎