One of the issues you may have run into if both Ysera and Dalaran are the same type of server (PvE, PvP, RP, RP-PvP) is that if you get phased to the same type of server, you typically won't de-phase once you leave group until you change zones unless the phase you were in is "full" (and there's no real way to figure out what counts as full that I've ever seen). For example: about a month ago now, I joined a group for a world quest in Azsuna; my character is on Korgath (PvP) and the group itself was on Ragnaros (also PvP, but one of the latin american servers). When I completed the world quest and left the group, I remained on Ragnaros—this was patently obvious because it was very quiet (which it never is on my CRZ because Blizzard has it out for me personally ;P) and everyone I ran past was from Ragnaros and had spanish guild names. I stayed on Ragnaros until I crossed the zone border into Val'Sharah twoish hours later, at which point I phased back into my normal CRZ.
So, if you initially put your group together in such a way as to bring your Yseran buddies to Dalaran, all subsequent attempts to get on Ysera would have failed; they would need to have changed zones to swap back to Ysera before you would have been able to try and get there again. Logging out and back in again may also have worked, but I'm not sure how long you have to be offline to lose your "spot" in any given partition; some people have said 5m is enough, others say it has to be a full 30.
I think the same logic also applies when you're swapping zones; if my friend on Stormrage invites me into a group and we're both in Highmountain, I will (usually) phase into his version of Highmountain. If we queue into dungeon finder, do a dungeon, and I leave the dungeon group first, both of us will end up on Korgath because I was the first one in the zone and even though he's leader, I "started" the group for the zone. Then he goes absolutely crazy cos even though he's on a PvE server he loves PvP and there's never a shortage of targets on my PvP CRZ because Alliance get outnumbered so badly, THANKS BLACKROCK >:( So, if you went into Duskwood before either of your friends, Duskwood would be on Dalaran even though your group has more people from Ysera in it.
Level also still plays a role—I'm pretty sure the higher level character's server still takes precedence—but as most of my experience with CRZ swapping comes at max level, I'm not super informed on the specifics there. Will a level 21 overrule two level 20s? I have no idea, more data is required. My hunch says that it needs to be a more significant level gap (~5 levels feels right) but I don't have a whole lot of facts to back up my gut on that one.
[ Basanti - Draconic Savant, A-Korgath(US) ]