I’m apparently a bit late as Tim has just answered the question but I’ll describe the procedure from the GUI shelf configuration (‘shelf -c’, available from at least 6.2.1).
I believe the confusion is that you are trying to add your application as a new type of shelf plugin. From the shelf configuration utility after you click on Before/After if you click Browse… to add your item it will only let you add a .so or .dll as you described.
Instead of clicking “Browse…”, you should select “Launch Application” in the “New Type:” list and click Apply. You’ll now have created a “Launch Application” item, and if you click on it you’ll see a box called “Program to Launch” to the right where you can select your application (or any file for that matter).
All the applications that are started from the shelf use this same procedure. Have a look at one of them, say the File Manager under Utilities, and you’ll see it’s name is “File Manager”, type “Launch Application” and Program to Launch is pfm. The Launch Application type is the launcher.so plugin that Tim discussed.