1- Linux VM system disks on VMWare Datastore and mailstore via direct linux iscsi initiator
2- Linux VM system disks on VMWare Datastore and mailstore on linux LVM of 2TB VMWare vmdk disks
3- Linux VM system disks on VMWare Datastore and mailstore on two VMWare RDMs
#1) leaves all the data on the NetApp, and gives plenty of options for doing things like netapp snaps, etc. Potentially more problematic from a multipathing scenario.
#2) LEaves everything in VMware, and gives the option of things like storage vmotion, etc. Potentially more pain with LVM, etc.
#3) Basically the same as #1, but leaving the multipathing to VMware. Difficult to do vmotions with, though.