I think the difference between an attraction that is fine like (me being attracted to women) and one that is not fine (dating a person only because they are disabled) is the basis of the relationship, not the attraction in particular. If I am dating a woman only on the basis that she has large breasts then our relationship is probably going to be crap but if I only consider that a bonus on top of an otherwise good relationship then everything is just fine. There are some really thorny problems though for those who are attracted to somebody who has a very rare trait. In the article I linked the person writing in was a lesbian who was only able to get about in a wheelchair - if that is who you are looking for you are going to have a very limited set of options for dating and the person in question would be right to wonder whether you are with them only because you can't find anybody else. It doesn't seem especially fair though to suggest that anyone who has a fetish for a rare trait is automatically suspect - there is no reason to give the stamp of approval to my desires just because they are statistically normal.
I also wonder if I am maybe looking at my own relationship incorrectly. Someone only interested in lesbians in wheelchairs would be viewed with some suspicion on the basis that if their partner were to be cured (of the disability, not the orientation!) they would lose interest. On the other hand I think people generally believe that if their partner were to declare that they were transgender that ending the relationship is reasonable. I am torn on that topic... I love Wendy in a multitude of ways that are gender irrelevant but would I still be attracted to her if she identified or physically transitioned to male? I feel like ending a relationship is acceptable in cases where your partner changes drastically who they are and gender is a pivotal part of our identities. I would certainly give the relationship a chance but I can't guarantee it would work; a relationship where I have no sexual desire for my partner seems untenable to me.