Camera Following Gripper

[kavidey]

I have a much better understanding of how the whole interface works and put together some prototype code for the camera to look at the gripper position. As of right now I have a toggle box to enable the camera follow mode that pings the backend every 200ms (similar to how we are doing continuous actions).

In order to test my code I need something to be publishing joint info in ROS, so have I switched gears a bit and have been trying to get the PickNik stretch gazebo simulation up and running. I'm happy to make a new github thread and track my progress on that too if it would be helpful.

[kavidey]

After researching /joint_state_publisher and /robot_state_publisher, I tried to get arm position data from /robot_state_publisher, but nothing was pushing data there even with the gazebo sim running (That may be because the gazebo simulation isn't working properly, but I'm not sure. See Issue #8 for more info)

I switched to getting arm position data from /tf, which seems to be working. I'm currently writing the trig math to convert arm positions into camera angles.

[kavidey]

While debugging gazebo, I realied that /tf is published by gazebo, so it would probably not be available on the actual robot. I'm looking into other transform sources. All the trig math updated interface are in gazebo_sim.

https://github.com/hcrlab/stretch_web_interface/blob/gazebo_sim/robot/ros_connect.js#L284
https://github.com/hcrlab/stretch_web_interface/blob/gazebo_sim/shared/commands.js

[mayacakmak]

Excellent @kaviMD! I think you were on the right track with joint_state or robot_state.. I wonder why they were not being published. Did you try something like $rostopic list and then go through the list to see if there's anything that might be what you're looking for? and then you can echo that topic to see what's in it. I think the robot is already doing forward kinematics internally so the joint poses should be published somewhere. We can definitely ask Charlie if you've exhausted options ;)

[kavidey]

This is fully functional and pushed now. Per our meeting tf is accessible on the robot, so I'm sticking with that to get the transforms. There was a git conflict earlier on this branch that had accidentally deleted the camera following code.

It turns out that to access transforms from tf in JS via roslib, you need to use a special TFClient, and have tf2_web_republisher running (I adjusted the launch file to start it automatically with the interface).

I think this is ready to merge into the main branch if we want to. Let me know if there is anything I can do to help debug it

[mayacakmak]

@kaviMD I just tested this on the real robot and it works very well and it is fun to use ;)

[mayacakmak]

Currently the 'look' actions are logged automatically but I think for when the camera is following the gripper we'll want to only log the start and end rather than continuously during the gripper following loop.

[kavidey]

With the update loop move from front end to back end in Issue #30, updating the logging should be relatively easy, if not handled automatically. What branch was that update pushed to?

[mayacakmak]

Ah yes, I updated that too, it just logs the start and the end ;) I just pushed. They're all in the upstream branch b/c I'm still not done testing those merges but issues from previous merges started coming up then. Will move to master soon.

[mayacakmak]

One thing I noticed while testing is that the camera following stops during continuous motions and catches up once the mouse is released--not sure why this is happening since they're in separate threads, but noting down here what I observed. Perhaps there is an obvious reason/fix.