vmfs3 Copy FIle Zero Bytes Copied SHell Hung - Debug Sequence

[aessatl]

I have been trolling for a user forum for making a post on this but did not find anything (active). As such maybe you can point me in the correct direction, or provide some ideas….

---

We have a lab, one of the more junior admins had a vSphere 4 farm. Took a vmfs v3 volume, single 300GB partition, and extended it via the iSCSI SAN controller to 2TB. This of course orphaned the vSphere 4 hosts from mounting the volume. I am trying to recover the data off the volume and move it onto a new VMFS 5 volume (had him upgrade his systems to vSphere 5). I was able to recover the first VM, but subsequent VMs have certain files which will not copy. The shell seems to “hang” with zero bytes transferred.

Steps:

1. Ubuntu VM with two virtual disks. One OS, One mounted for repository which will later be exported as NFS for the vSPhere 5 systems to mount.
2. Mount original vmfs3 volume via iSCSI and run a basic cp command to transfer the files from the vmfs3 volume to the local formatted virtual disk which is exported via NFS.
3. Via network, transfer from “NFS” volume that the new vSphere host mounts, to the new VMFS5 volume.

root@ubuntu12:/home/ibm/vmfs-tools-0.2.5# dpkg -s vmfs-tools
Package: vmfs-tools
Status: install ok installed
Priority: extra
Section: otherosfs
Installed-Size: 232
Maintainer: Ubuntu Developers [email protected]
Architecture: amd64
Version: 0.2.1-1
Depends: libc6 (>= 2.7), libfuse2 (>= 2.8.1), libuuid1 (>= 2.16)
Description: Tools to access VMFS filesystems
VMFS is a clustered filesystem designed to store virtual machine disks for
VMware ESX or ESXi Server hosts. This set of tools allows to access these
filesystems from some other non ESX/ESXi host for e.g. maintenance tasks.
.
Only read access is available at the moment, but write access is under
works. Multiple extents are supported.
.
The VMFS can be accessed with a command line tool or mounted through a
userspace filesystem (FUSE-based).
Original-Maintainer: Mike Hommey [email protected]
root@ubuntu12:/home/ibm/vmfs-tools-0.2.5# mount
…
/dev/fuse on /media/vmfs type fuse (rw,nosuid,nodev,default_permissions)
/dev/sdb1 on /media/floppy0 type ext3 (rw)

root@ubuntu12:/home/ibm/vmfs-tools-0.2.5# exportfs
/media/floppy0 .
root@ubuntu12:/home/ibm/vmfs-tools-0.2.5# ls -alh /media/vmfs/hpdc02/
total 0
drwxr-xr-x 2 root root 2.8K Mar 8 13:50 .
drwxr-xr-t 36 root root 5.7K Jun 29 11:10 ..
-rw------- 1 root root 2.2G Jul 7 19:46 hpdc02-000001-delta.vmdk
-rw------- 1 root root 321 Feb 8 2012 hpdc02-000001.vmdk
-rw-r--r-- 1 root root 37 Mar 8 13:50 hpdc02-606787e8.hlog
-rw------- 1 root root 4.0G Jul 7 15:46 hpdc02-606787e8.vswp
-rw------- 1 root root 35G Feb 8 2012 hpdc02-flat.vmdk
-rw------- 1 root root 8.5K Apr 26 10:15 hpdc02.nvram
-rw------- 1 root root 4.1G Feb 8 2012 hpdc02-Snapshot1.vmsn
-rw------- 1 root root 522 Dec 17 2010 hpdc02.vmdk
-rw-r--r-- 1 root root 422 Feb 8 2012 hpdc02.vmsd
-rwxr-xr-x 1 root root 3.2K May 4 10:59 hpdc02.vmx
-rw-r--r-- 1 root root 261 Dec 28 2011 hpdc02.vmxf
-rw-r--r-- 1 root root 137K Jul 11 2011 vmware-30.log
-rw-r--r-- 1 root root 60K Jul 11 2011 vmware-31.log
-rw-r--r-- 1 root root 130K Aug 23 2011 vmware-32.log
-rw-r--r-- 1 root root 128K Oct 11 2011 vmware-33.log
-rw-r--r-- 1 root root 128K Dec 5 2011 vmware-34.log
-rw-r--r-- 1 root root 324K Mar 2 14:33 vmware-35.log
-rw-r--r-- 1 root root 205K May 11 11:31 vmware.log
root@ubuntu12:/home/ibm/vmfs-tools-0.2.5# ls -alh /media/floppy0/hpdc02/
total 14G
drwx------ 2 root root 4.0K Aug 23 13:29 .
drwxr-xr-x 6 root root 4.0K Aug 30 11:20 ..
-rw------- 1 root root 0 Aug 23 11:59 hpdc02-000001-delta.vmdk
-rw------- 1 root root 321 Feb 8 2012 hpdc02-000001.vmdk
-rw-r--r-- 1 root root 37 Mar 8 13:50 hpdc02-606787e8.hlog
-rw------- 1 root root 128K Aug 23 12:59 hpdc02-606787e8.vswp
-rw------- 1 root root 35G Feb 8 2012 hpdc02-flat.vmdk
-rw------- 1 root root 8.5K Apr 26 10:15 hpdc02.nvram
-rw------- 1 root root 0 Aug 23 13:23 hpdc02-Snapshot1.vmsn
-rw------- 1 root root 522 Dec 17 2010 hpdc02.vmdk
-rw-r--r-- 1 root root 422 Feb 8 2012 hpdc02.vmsd
-rwxr-xr-x 1 root root 3.2K May 4 10:59 hpdc02.vmx
-rw-r--r-- 1 root root 261 Dec 28 2011 hpdc02.vmxf
-rw------- 1 root root 0 Aug 23 13:29 vmware.log
root@ubuntu12:/home/ibm/vmfs-tools-0.2.5#
root@ubuntu12:/home/ibm/vmfs-tools-0.2.5# cd /media/
root@ubuntu12:/media# ls
cdrom floppy0 hp nfs vmfs
root@ubuntu12:/media# chmod 770 /media/vmfs/hpdc02/vmware.log
root@ubuntu12:/media# ls -alh /media/vmfs/hpdc02/vmware.log
-rw-r--r-- 1 root root 205K May 11 11:31 /media/vmfs/hpdc02/vmware.log

<…hmm… chmod not working as expected…>
root@ubuntu12:/media#
root@ubuntu12:/media# cp -a /media/vmfs/hpdc02/vmware.log /media/floppy0/hpdc02/

<>> Review via other shell below.
Before Shell cp command started After / During (as shell becomes orphaned)
root@ubuntu12:# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ubuntu12-root 15400892 1322480 13305704 10% /
udev 500692 4 500688 1% /dev
tmpfs 203900 248 203652 1% /run
none 5120 0 5120 0% /run/lock
none 509748 0 509748 0% /run/shm
/dev/sda1 233191 24965 195785 12% /boot
/dev/fuse 1610350592 825751552 784599040 52% /media/vmfs
/dev/sdb1 309633052 29085696 264818920 10% /media/floppy0
root@ubuntu12:# root@ubuntu12:# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ubuntu12-root 15400892 1322480 13305704 10% /
udev 500692 4 500688 1% /dev
tmpfs 203900 248 203652 1% /run
none 5120 0 5120 0% /run/lock
none 509748 0 509748 0% /run/shm
/dev/sda1 233191 24965 195785 12% /boot
/dev/fuse 1610350592 825751552 784599040 52% /media/vmfs
/dev/sdb1 309633052 29085696 264818920 10% /media/floppy0
root@ubuntu12:#

(( bytes do not change so shell seems to be in some kind of unknown state))

Are there any tools or ideas on how to further debug this so I can recover these files. Obviously without these files, I cannot recover the VM(s). It worked without issue on hpdc01 but on subsequent VMs file copies have all had one or more failures, but not yet seeing pattern.

Thanks,