Are you working too add any of these two desktops ? I try "Debian Wheezy armHF 1.0 beta" and it runs really smoother if I have to compare with "Linaro Ubuntu 12.04 LTS 2.0" !! Cheers
Should be as simple as apt-get remove gnome and then apt-get install xfce or whatever environment you want. Thanks, EBrown