I have a Westley Richard's patent 1871 rifle. It has a 500 No. 2 Express chamber.  It has a Whitworth rifled bore-8 flats, octagon.   

This is a cartridge gun, not a ML.

The  bore slugs .502 across the flats (minor diameter) and .527 across the corners (major diameter).

I have a round grease groove bullet mould for .527 bullets weighing 487grs in 20-1.

I am reluctant to shoot a .527 diameter bullet, even 20-1 lead down .502 minor diameter bore.

Does anyone have any experience or knowledge to share on shooting Whitworth bore cartridge guns?

Why not try a 50-70 bullet (.512ish) at 450 to 500 grains and see how they work? You can always try wrapping them with increasing layers of paper patch until it all comes together.

My 1869 WR Improved Martini is in 500-450 #2 Musket and is marked Henry rifling. The Imperial War Museums 1871 WR Improved Martini (outside the same as my 69 but had gone to a striker) is marked "for Martini cartridge" which I assume is the 577-450 and the top of the barrel is marked Metford rifling. These are sturdy actions which can be found in many calibers and, apparently, with many barrel forms. Our BP lead shooting rifles are starting with bullets a few thousandths over groove diameter no matter what the bore diameter and the Whitworth while mostly bore diameter in appearance actually has triangular grooves which could possibly be measured from the sharp point of the intersection of two flats to the distance of the  center of the line drawn between the centers of two flats. Sure wish I could draw that. The bottom line is that the bullet should be of a larger diameter than the apparent bore diameter. I'd pop a 50-70 (say .512 diameter) into the barrel using just a Magnum primer and then drive it back out from the nose side and see what it looks like.
I was just looking around and found this: The .577/500 No. 2 BPE was developed as a black powder round some time before 1879 by necking down the .577 Black Powder Express to .507-inches (12.9 mm)[2] for use in single or double rifles, as well as a variety of Martini-based lever rifles. Now you can imagine that some hunter in the middle of nowhere ran out of his relatively rare Whitworth cartridges so he fired off a few of these at the charging Cape buffalo with no ill effects but perhaps not the best accuracy due to the bullet being too small. An experiment is in order.
Update - I just found your photo of the rifling in your barrel and would call it Metford polygonal. Shallow with rounded lands. Treat it like any BP rifle with a bullet several thousandths larger than the land diameter.