The Park51 Community Center organizers have an official website:
http://www.park51.org/facilities.htm Here is what they say:
Park51 will grow into a world-class community center, planned to include the following facilities:
•outstanding recreation spaces and fitness facilities (swimming pool, gym, basketball court)
•[...]
•a mosque, intended to be run separately from Park51 but open to and accessible to all members, visitors and our New York community
•[...]The reason opponents say they are opposed to "the mosque" and not "the community center" is because they are not opposed to the swimming pool, or the gym, or the basketball court. They are opposed to the mosque.