Error : Check failed: error == cudaSuccess (30 vs. 0) unknown error

[rahulbhalerao001]

I had created a private AMI from a running code (after the cache changes), and the Imagenet example was running correctly on this AMI.

However, today I created a new cluster from this AMI and got the error - "Check failed: error == cudaSuccess (30 vs. 0) unknown error".

[robertnishihara]

Did you get the error when running ImageNetApp.scala? Was that the only difference? What sort of nodes were you using, and what OS? Were you using the current SparkNet master, or did you modify anything?

[rahulbhalerao001]

Yes Imagenet.scala. After posting, I pulled, rebuilt, and re ran with same result.
Node - g2.8xlarge with ubuntu 14.04 (same as the AMI provided here)
No, I did not modify.

[rahulbhalerao001]

I made a private AMI, because I wanted to stick to one version of the code.

[robertnishihara]

I'm not sure exactly what the problem is, but one starting point is to figure out exactly where the error is occuring. A couple ways to do this:

1. Start a spark shell with something like ~/spark/bin/spark-shell --jars /root/SparkNet/target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar and try loading a model from a model file and creating a net and calling net.forward, and see precisely where it crashes. For this purpose, you can do all of this on the Spark master and you don't need to create a net on each worker.
2. Sometimes running things in the Spark shell is different from running a script, so I'd suggest commenting out components of CifarApp.scala until you stop getting the error to find the minimal example that causes it to fail.

By the way, you said the problem was with ImageNetApp, but does it also occur with CifarApp?