Trouble setting up native CSI raspberry pi camera ( IMX708 ) and some other small questions.

[Davidsastresas]

Hi!

I've been playing around with this project for some days, first within the same desktop machine, and now I set it up in a raspberry pi compute module 4 board, using a ochin v2 carrier board (https://github.com/ochin-space/ochin-CM4v2) a very nice board for this project by the way.

I managed to build from source in the raspberry pi itself, just for the fun of it, and it took forever, more than 1 hour. Probably cross-compiling from Desktop is nicer for embedded development.

Now I have the setup working, consisting on raspberry pi cm4 connected on its serial0 to Ardupilot SITL ( through an FTDI ), and then by ethernet connected to QGC on my desktop machine. I manage to have the ball test pattern working in QGC, handshake and video autodetection working properly.

I am not managing to use imx708 ( raspberry pi camera v3 ). It works nicely using rpicam-hello and gstreamer. These are the devices detected by v4l2:

➜  ~ v4l2-ctl --list-devices                                                                                   
bcm2835-codec-decode (platform:bcm2835-codec):
        /dev/video10
        /dev/video11
        /dev/video12
        /dev/video18
        /dev/video31
        /dev/media3

bcm2835-isp (platform:bcm2835-isp):
        /dev/video13
        /dev/video14
        /dev/video15
        /dev/video16
        /dev/video20
        /dev/video21
        /dev/video22
        /dev/video23
        /dev/media2
        /dev/media4

unicam (platform:fe800000.csi):
        /dev/video0
        /dev/video1
        /dev/media0

rpivid (platform:rpivid):
        /dev/video19
        /dev/media1

which seem to be reflected too in the configuration tool:

I also receive a bunch of errors like this for most of /dev/video* when starting from command line with --verbose argument:

2024-08-29T19:00:52.057244Z  WARN             v4l_wrap ThreadId(14) src/lib/video/local/video_source_local_linux.rs:738: Failed to capture formats for device: "/dev/video13". Reason: Os { code: 22, kind: Invalid
Input, message: "Invalid argument" }                  

I saw some issues resolved about this and my understanding is that native Rpi camera support was functional. Anybody has any hint to keep debugging?

Also, I am experiencing some small issues, not super important at this point, but just for awareness in case anybody has any input:

• When changing configuration, specifically mavlink endpoint, as arguments when launching mavlink-camera-manager, it doesn't update unless I launch 2 times. First time I run it with the updated argument it still uses old --mavlink argument. This one is not a big deal.

• When connecting to the autopilot, and GCS, I am finding it doesn't always connect, sometimes I need to reboot several times for the whole thing to work and for mavlink-camera-manager to do the handshake alright with QGC. I am using QGC 4.4 and enabling cameramanager logs, and it seems it asks for camera info a lot of times until giving up, and then a few seconds later, sometimes that camera info is received and the procedure continues as usual until video configuration is digested by QGC and video shows up, but other times it does not receive it and it doesn't proceed further.

Mavlink traffic seems to be clean, no errors at all deducted from sequence number. Usually I have better luck if I start Ardupilot first, and then mavlink-camera-manager. If starting mavlink-camera-manager first, it is more likely that the handshake won't complete from my experience.

Is there anything in particular that is tricky on this part? related to mavlink system ID, or timing on this handshake, etc?

Thank you very much!

[joaoantoniocardoso]

Hi @Davidsastresas, welcome!

I managed to build from source in the raspberry pi itself, just for the fun of it, and it took forever, more than 1 hour. Probably cross-compiling from Desktop is nicer for embedded development.

you can build it for your architecture by using our cross docker w/ gstreamer by invoking cross using the target argument of cargo-cross, as in cross build --release --locked --target armv7-unknown-linux-gnueabihf. The supported architectures (so far, it is easy to expand) are in the Cross.toml file. You can get cargo cross by means of cargo install cross.

I am not managing to use imx708 ( raspberry pi camera v3 ). It works nicely using rpicam-hello and gstreamer. These are the devices detected by v4l2:

Unfortunately, MCM doesn't yet support rpi cam v3. I have just gotten one myself, and I'll try adding support for it ASAP. This issue is essentially what we need to have it working.

When changing configuration, specifically mavlink endpoint, as arguments when launching mavlink-camera-manager, it doesn't update unless I launch 2 times. First time I run it with the updated argument it still uses old --mavlink argument. This one is not a big deal.

This is indeed a known problem that hasn't yet been addressed, we should make the CLI to always have priority over the config file.

When connecting to the autopilot, and GCS, I am finding it doesn't always connect, sometimes I need to reboot several times for the whole thing to work and for mavlink-camera-manager to do the handshake alright with QGC. I am using QGC 4.4 and enabling cameramanager logs, and it seems it asks for camera info a lot of times until giving up, and then a few seconds later, sometimes that camera info is received and the procedure continues as usual until video configuration is digested by QGC and video shows up, but other times it does not receive it and it doesn't proceed further.

I've seen this happening, I'm unsure if the problem is in the MCM or QGC. Does it happen in QGC v4.3.0 as well?

Thanks!

[Davidsastresas]

Thank you very much for your answer, and for the tip about cross-compiling, much appreciated.

About Rpi cam v3, if I understand correctly then, including some custom gstreamer that has libcamera operational should do the job? Or besides that is there some additional modification we would need to do over MCM code itself?

We were considering using this project for a production environment, and for the short future we wanted to work with imx708 and a Toshiba tc3587 based HDMI to CSI converter to be able to use HDMI cameras, using both of them at the same time to be able to switch video sources from a custom QGC build.

About where we tested, what I reported is true for latest QGC 4.4. I will try to gather more information about this first connection issue and report back if I find something worth mentioning.

Thank you very much!

[joaoantoniocardoso]

About Rpi cam v3, if I understand correctly then, including some custom gstreamer that has libcamera operational should do the job? Or besides that is there some additional modification we would need to do over MCM code itself?

Besides letting the libcamerasrc GStreamer plugin be available in your environment, there is an integration in the code to be done. The work starts with adding a new enum variant to lib::video::local::VideoSourceLocalType and filling the gaps to bridge it with the rest of the code. The uncertainty is that GStreamer often requires some tweaking regarding video capabilities. Let me know if you are interested in collaborating with that.

Thanks!

[Davidsastresas]

Thanks for the answer.

We need to have something functional a few weeks from now. The easy solution is to hard code some scripts to use our required imx708 and tc3587, but I certainly would prefer to use MCM. So I will take some weeks to try to add support for it, specially as it seems it could be interesting for the project too! And if I run out of time I will come back to the less nice plan of hard coding those scripts.

I am kind of familiar with Gstreamer at the basic level, at integrating it in other projects, but I have never actually got involved in Gstreamer development. I have never worked in Rust before either. I am quite comfortable in c/c++, and not so comfortable but I can manage in python. So if that is fine I will probably ask some questions in the process.

So, if I understood correctly, the process right now to keep pulling this string would be:

• Download/build a gstreamer version that has libcamerasrc. About this, is there any part of MCM that needs to specify Gstreamer version, or that is only needed to specify in CI files for build pipelines?
• After the above, pull that string of lib::video::local::VideoSourceLocalType.
• Possible trouble in gstreamer - After the above is done I will report back to see how the situation is on this front.

About the part of adding support for tc3587 too, is there something on top of your mind right now that could help me get on track faster?

Thank you very much!

[joaoantoniocardoso]

We need to have something functional a few weeks from now. The easy solution is to hard code some scripts to use our required imx708 and tc3587, but I certainly would prefer to use MCM. So I will take some weeks to try to add support for it, specially as it seems it could be interesting for the project too! And if I run out of time I will come back to the less nice plan of hard coding those scripts.

I am kind of familiar with Gstreamer at the basic level, at integrating it in other projects, but I have never actually got involved in Gstreamer development. I have never worked in Rust before either. I am quite comfortable in c/c++, and not so comfortable but I can manage in python. So if that is fine I will probably ask some questions in the process.

Sounds fair. I can work with you in that process so we lower the risk. Rust is not that different from C/C++. It is less flexible, and everything is done by composition. To help you focus on what is important for this work, MCM is heavily based on the Tokio async framework, and the GStreamer rust bindings. This is the default book people use to learn it, but there is also this other one, which is way more direct, but very shallow.

Download/build a gstreamer version that has libcamerasrc. About this, is there any part of MCM that needs to specify Gstreamer version, or that is only needed to specify in CI files for build pipelines?

Yes, I believe we can use Ubuntu's gstreamer1.0-libcamera. Debian probably has it as well. What SO are you using? There is a Dockerfile here in the repository that can be a starting point. If GStreamer is not available with libcamera in your platform, we have this script that can serve as a guide.

After the above, pull that string of lib::video::local::VideoSourceLocalType.

Yes.

Possible trouble in gstreamer - After the above is done I will report back to see how the situation is on this front.

I'd prefer if you instead start by creating a draft PR here and update it whenever you want to discuss the things, if you think it works for you.

About the part of adding support for tc3587 too, is there something on top of your mind right now that could help me get on track faster?

We'd have to do some research on that, maybe it's also using libcamera?

Thanks

[Davidsastresas]

Thank you very much for the response, especially for those resources to start with rust. I already started reading them, and in the following weeks I hope to make advance.

I will close this issue and make a PR when I start working on it, to continue the discussion there.

thank you!