Clairify/decide how/where we will install condor/bosco (Dockerfile)

[bdklahn]

Running the following,

cloud-copasi/Dockerfile

Lines 36 to 37 in 0376aa9

	USER cloudcopasi
	RUN ./condor_install --bosco

. . . in a recent tar.gz version extract location (e.g. /condor) . . .

cloud-copasi/Dockerfile

Line 32 in 0376aa9

ENV condor_version="9.8.0" condor_build="20220217"

. . . seems to default to installing a bosco into the user's home directory.
e.g.
/home/cloudcopasi/bosco

Are we not using the --bosco flag install? Do we need it to be /home/cloudcopasi/condor?
-or was that where the tar.gz source installation files were being extracted to?
-or were we using one of the deb or rpm package installs?

[bdklahn]

It sounds like, via @hasanbaig (Thanks!), condor_install --bosco is now a bit depreciated. (Anyway, that install process seems to have some bugs, indicating it may not be maintained.)

So I'll plan on just extracting the appropriate tar.gz for "Linux user install", into /home/cloudcopasi/condor. Then if the condor_config ENV is set right (#21), hopefully things can work.

[hasanbaig]

Pasting here again:

1. extract and install in condor/ directory.
2. copy our custom bosco_cluster file into condor/bin/ directory.
3. add FS_LOCAL_DIR=/home/cloudcopasi/condor/local/fs_auth line in the "~/condor/local/config.d/condor_config.local" file.

[bdklahn]

Ha! @hasanbaig I was about to paste that in here myself! Great minds think alike!

So on that third point: It looks like I have to mkdir -p ~/condor/local/config.d/ (and can create that file implicitly, by echoing that line into it). Because those don't already exist in the installed condor directory.
Is that right?
As well, it looks like fs_auth doesn't yet exist.

I could also try and create/fill that in the Docker file, if possible, if you let me know.

[hasanbaig]

I don't remember exactly if the directory config.d was already there or we created it. If it does not exist in the fresh installation then yes we have to create it.

And yes, fs_auth has to be created.

[hasanbaig]

I think fs_auth directory should be there already.

[bdklahn]

There is no condor/local directory at all, in/from the ENV condor_version="9.8.0" condor_build="20220217" . . . x86_64_Ubuntu20-stripped.tar.gz file.

root@74d171b8bd18:/home/cloudcopasi/condor# ls -1
'*_local_submit_attributes.sh'
LICENSE-2.0.txt
Manifest.txt
NOTICE.txt
bin
condor_configure
condor_install
etc
examples
include
lib
libexec
man
sbin
usr
var

https://research.cs.wisc.edu/htcondor/tarball/current/9.8.0/daily/condor-9.8.0-20220217-x86_64_Ubuntu20-stripped.tar.gz

I can try the older version you were using (9.1.2?).

[hasanbaig]

Okay. I have just looked up in my past logs and emails and below is the quote from one of his (HTCondor's developer whom we took on board to debug some issues) email:

"For the HTCondor installation under ~/condor, we recommend putting your config changes in files under ~/condor/local/config.d/. Then, any time you need to upgrade or re-install, you just need to save the contents of that directory to maintain your configuration."

I remember that it was him who created this folder called local for performing some bosco-related tests (we gave him an access to do so) to debug some issues related to bosco_test pool.

We then, on his own suggestion, created config.d/condor_config.local in the same local directory.

[bdklahn]

Ah. Ok. Thanks!
I can do that in the Dockerfile.
Is there anything we need to put into fs_auth, or is it just empty?

[hasanbaig]

No, I think you can continue with the latest version. It took me a little while to look back into past stuff to recall and reply :-).

[hasanbaig]

Ah. Ok. Thanks!
I can do that in the Dockerfile.
Is there anything we need to put into fs_auth, or is it just empty?

It is Empty.

[bdklahn]

No, I think you can continue with the latest version. It took me a little while to look back into past stuff to recall and reply :-).

Cool!
No worries . . . I'd imagine there was a lot of stuff you went over with them. And this whole configuration/install is fairly complicated. Hopefully, with a carefully made Dockerfile, version tagging (and eventual GitHub Actions builds), maintaining and updating will be a little more systematic and easier.

[hasanbaig]

🤞

[bdklahn]

Okay. I have just looked up in my past logs and emails and below is the quote from one of his (HTCondor's developer whom we took on board to debug some issues) email:

"For the HTCondor installation under ~/condor, we recommend putting your config changes in files under ~/condor/local/config.d/. Then, any time you need to upgrade or re-install, you just need to save the contents of that directory to maintain your configuration."

I remember that it was him who created this folder called local for performing some bosco-related tests (we gave him an access to do so) to debug some issues related to bosco_test pool.

We then, on his own suggestion, created config.d/condor_config.local in the same local directory.

Hmmm? If we do have any other config changes, we should probably save and track them in this repo. Then just COPY them over itemwise (file or subdirectory), rather than baking all the customization into the Docker file (Although the Dockerfile is also tracked, etc.).

Perhaps it would be good to convert our bosco directory into a condor directory, with just our changes included, in a directory structure mirroring the stock condor install directory.
e.g.

> tree condor
condor
├── bin
│   └── bosco_cluster
└── local
    ├── config.d
    │   └── condor_config.local
    └── fs_auth

[hasanbaig]

I just checked your last comment. Yes, we can do it this way.

[bdklahn]

I pushed a condor_overlay directory, with corresponding changes in the Dockerfile, and deletion of the bosco directory.
3147505