The National Portrait Gallery sounds like the least of Washington, D.C.’s offerings. You might think: a bunch of oil-painted headshots of some glum people who never smile because they have bad teeth. Oh look, it’s Cassius P. Brandograst, Deputy Postmaster General in 1827. Can’t get that back home.

That’s what you might think — and you’d be wrong.

The Portrait Gallery might be the capital’s most surprising museum. Diverse, brilliantly curated, with an added bonus of some remarkable architecture that most visitors never see.

You can’t blame the visitors for passing it up. The big-name museums of Washington get all the business, because they’re spectacular family-friendly theme parks like the Air and Space and Natural History museums. The Holocaust Memorial and African American History museums provide somber history lessons no one should ignore. For a binge-’til-you-sag tour of the wonders of Western Civilization, there is the National Gallery of Art. Whatever you choose, you’ll run out of day before you run out of things to see.

But what if you’re not in the mood for something huge? That’s where the fourth type shines. The Folger Shakespeare Library, the Hirshhorn, the Phillips Collection, the Hillyer Art Space — and the Portrait Gallery.

There might be a pedant who hears you say “I went to the Portrait Gallery” and replies “actually, it’s the Patent Office Building, and it houses the Portrait Gallery and the Smithsonian American Art Museum.” This person is very dull and can be ignored.

Rather than cite a few notable pieces, it’s best to just say what you can expect: everything.

Outsider art, such as a huge altar made from light bulbs and aluminum foil by a janitor in his garage? Yes. Gorgeous and heartbreaking 19th-century sculpture by American masters? Yes. Little-known early-20th-century representational art from painters making a new pictorial style? Yes. Immense and captivating video installations, Depression-era urban landscapes. A few examples of tiresome modern art; photographs from the collection of Mathew Brady and Irving Penn.

And portraits! Presidents galore. Nixon, looking like he’s trying to humor some rich donor describing an acid trip; stolid Coolidge, haughty Wilson, the floral exuberance of the new Obama portrait.

You can do it in a couple of hours tops, and you’ll see things you might walk by in a big museum because there’s a famous painting you feel obligated to observe.

The art is only part of the appeal. The building itself is a history lesson. Construction began in 1836, and continued for 31 years. It would have taken less time to haul a block-wide hunk of granite to the site and whittle it down. One of the few Greek Revival buildings in town, it presented a sober retort to the opulence of the Roman styles the politicians liked. A serious building for the serious matter ... of patent models.

The law required patent applicants to submit a working model of their idea, proving it actually worked. These models were on display on the third floor of the Patent Office, and citizens could wander around, studying the newfangled ingenuity. During the Civil War, the building was used as a hospital for the wounded; Walt Whitman, who thought the building was the “noblest” of D.C. structures, showed up to read aloud to the wounded men. (Whitman worked in the building for a while, but was fired — supposedly for having a naughty book in his desk drawer. Bonus: It was his own work, “Leaves of Grass.”)

The building has been restored to pristine condition, after years of work and almost a third of a billion dollars. You won’t find a more unique space than the Model Hall, a riot of colored tile and Industrial Age design, a place of grace, beauty and confidence.

Naturally, someone wanted to tear it down.

In 1952 local businesses wanted a parking ramp, and petitioned some pliable agencies and politicians to get a bill authorizing the building’s destruction. It didn’t pass. The man who sponsored the bill probably has his portrait on file somewhere in the basement; it doesn’t seem to be on display.

Perhaps it is, somewhere. The public can’t use staff bathrooms, so it’s difficult to know.