For me, Bruce's speech was definetly the highlight of SXSW. There were also exotic panels that I found interesting, like open sourcing science (especially after having recently watched The Corporation).
There is a core group of people who attend that are focused on social issues, while some others are more focused purely on gaming, technology and AJAX type issues (making the web more interactive and more interesting). You can also sense a bit of overlap between those groups.
I also think there are a bunch of people a bit new to the web looking to get their feet wet. I sorta felt that for that group it was similar to me going to my first SES about 3ish years ago, but attacking the web from a different angle.
Many of the panels were focused on blogs and blogging business models, and I really saw no need to attend most of those since I feel I read that sort of stuff almost every day. There are also some on programming. I would love to learn more about programming, but it is hard to balance doing everything I want to.
Austin is a cool town, and the weather is a nice break from winter. A couple blocks from the conference there is a river that winds through the town. You can walk down by the river for miles. The conference doesn't start until 10AM and I kept waking up early, so I went for a morning walk almost every day.
There are multiple parties every night, but I chose the morning walk over going out to the parties. One night I did enjoy a kick ass bullshitting session with Andy, and then we ate awesome Japanese food. Much to my surprise Andy said I was good with chop sticks.
There were lots of cool people there, but I am not a very social creature. If you know who the most popular bloggers and web designers are you keep running into them. For example, in the airport alone I think I saw Zeldman (probalby the most popular web designer in the world), Veen (the lead guy Google hired with MeasureMap), and on the plane I sat next to a couple people who worked at AOL.
If one was more business and networking oriented than I am they could probably find some top noche people to partner up with on projects - from hungry people new to the market right up to some of the best developers in the world.
Tim Mayer and Andy Hagans shared a panel on search, but largely due to the audience, that panel was very intro level.
If you like music and can stay an extra week there is a huge indie music festival that starts right around when the interactive / internet part of SXSW ends. Here is my brief review of 2005 SXSW.