The people at National Geographic tried it and it worked!! So cool!! It was actually done in part of a new National Geographic Channel series called How Hard Can it Be?, which will premiere in fall 2011.
A team of scientists, engineers and world-class balloon pilots successfully launched a 16'x16' house 18' tall with 8' coloured weather balloons and set a new world record for the largest balloon cluster flight ever attempted. 300 helium balloons were used! The lightweight building reached more than 3000m and remained in the air for about an hour.