hi
im putting this here for anyone in the future who needs a solution for
using your own domain name as ns 1 and ns2.
what worked for me
during the installation of imscp i added the 2 ip addresses which came with the vps.
after the installation i created the main domain then check both ip addresses to be used with it
created and configured an email
i needed a PTR record because without it i was able to send emails but not receive emails. for me i add to wait 3 days for it to fully propagate. the PTR record was set by my vps provider
another issue is if other mail servers cannot obtain a reverse lookup on your domain name then your new server will not get emails, or possibly be listed in the spam register. (so be careful when sending test mails,,, especially to google.. they will send you warnings in you mail.log)
on this installation i chose not to use policy-weight as i noticed that some times the service would die which caused problem.
anyway i hope this will help some one who is not an expert on dns like myself.