The blaze at the Plymouth Hall Satellite center in North Philadelphia was reported shortly before 8 p.m. Thursday and was brought under control a little more than an hour later.
Flames were visible and smoke poured from the four-story building as firefighters went inside to rescue some of the more than 60 residents.
A spokesman for Temple University Hospital declined to identify the victims.
Dozens of displaced residents were taken to a nearby apartment house while others waited for relatives aboard a parked city bus.