Dynamic CRM's Address Book is broken

I discovered this about 6 months ago and I'm yet to see a fix for it. The theory goes that if you're running the Outlook client and you're writing an email, you can click, say, the 'To...' button, go to the CRM Contacts address book and add contacts directly to your email.

It almost works. If in the CRM options, on the address book tab you opt to only sync the contacts that you own to the address book, you will see a big fat nothing. A few of the other address books populate such as the CRM Users but nothing in the Contacts address book.

The only way to make it work is to sync down ALL contacts in CRM. This can be problematic if your CRM DB has millions of Contacts but, for the moment, this is the only option you have.