I bet your TV is converting the (non-HD) multichannel DD/DTS bitstream from the HDMI link from the 360 into 2ch PCM on the optical output. I think I have seen someone mentioned that over at AVSForum a while back. That's why your "5.1" indicator on the receiver did not turn on because the receiver is not getting the discrete multichannels audio stream anymore.
You do have 2 options:
(1) Buy a receiver that can decode audio stream from HDMI (must be HDMI 1.1 or later). Here's is a catch: Xbox360 cannot pass through the latest HD discrete multichannel bitstream like Dolby Digital Plus, DTS HD Master Audio, etc. via its HDMI output for its HD-DVD add-on. However, gaming should be fine. Since I don't have a HDMI 360 (mine is old) check with the folks at AVSForum, etc.
(2) Bite the bullet and get the M$ HDMI cable kit which included a dongle with optical SPDIF output like your component cable. The dongle can be plugged in together with the HDMI cable. However, I haven't seen one in store... but then I haven't really looked for it so I could be wrong. Or, get the Elite which includes this dongle
