Hi Santosh,
You can open custom form on click of ribbon button. You have to provide the JavaScript library inside the Action of Command definition. You can do it through Workbreanch Ribbon editor or manually.
<Actions><JavaScriptFunction
Library=“$webresource:new_MyJavaScript“
FunctionName=“OpenCustomForm“>
</JavaScriptFunction>
</Actions>
Refer below blog.
http://nishantrana.wordpress.com/2010/11/04/using-javascript-in-a-custom-button-in-crm-2011/
You will have to construct the URL in JavaScript function that you want to open and pass
extraqs that will be fields that you would like to set.
Refer below MSDN blog for this.
http://msdn.microsoft.com/en-us/library/gg334375.aspx
Hope this helps!
Thanks!