IN DOWNTOWN SAN Francisco, a block can make a big difference. Walk just five minutes from Union Square and its palm trees and open-top tour buses; past designer shops and cable car stops; beyond the Hilton, the car hire storefronts, and where you rent bikes to ride over the Golden Gate Bridge. Then O'Farrell and Taylor streets meet. And San Francisco as you knew it disappears.

Welcome to the Tenderloin: the city's most notorious neighbourhood. Clusters of homeless people crowd pavements; others lie passed out in the street. Drugs are dealt in broad daylight. It's even said the Tenderloin's sketchy reputation gave it its name: that police officers were paid more to brave working here, and so could afford better cuts of meat. The typical hotel concierge will cross it off your map – perhaps even in histrionic red ink.

This might not seem like the obvious choice for a rapid new influx of hot boutique hotels, restaurants and bars. A courageous few arrived early: Brenda's French Soul Food, easily San Francisco's best brunch spot, with the queue to prove it; Whitechapel, a London Underground-themed gin bar; and Jessica Silverman Gallery, an airy, all-white contemporary art space occupying the ground-floor of low-income housing. Now, they've got company. Most recently, Gibson appeared on Eddy, one of the Tenderloin's seediest streets. The Art Deco-style restaurant serves oh-so-on-point sharing plates like warm bone marrow flan with wild nettle, and smoked duck with huckleberry.

But this isn't a gentrification story. Not in the sense of Brooklyn or East London, where the existing residents are pushed out to less-desirable fringes, dispossessed of their new-and-improved neighbourhood. In the 1980s, activists fought for laws that decreed the Tenderloin's low-income housing can never be torn down or snapped up by greedy luxury developers. Today, around 25% of housing here is run by nonprofits. This is where San Francisco's dependent poor, senior and disabled people live. And they can't be kicked out. They would, however, probably appreciate a reinvigorated place to live.

HIDING IN PLAIN SIGHT

Interior at the Gibson, Tenderloin, San Francisco
Drinks at Gibson, Tenderloin, San Francisco

The Tenderloin is both as bad as, and much better than, is commonly believed. Popular discourse paints it as riddled with violence and crime. But, truth be told, I've never felt unsafe here. You are more likely to feel dispirited than scared. Mental illness, poverty and squalor are rife and apparent. Fuelling this rot is that it's an open-air drug market. I've seen addicts shoot up in sun-lit doorways; walked past people openly arguing the price of a rock on a warm afternoon.

"No city official, including the Chief of Police, will acknowledge they're doing anything to keep drugs in the Tenderloin," says Randy Shaw, a fixture around these parts. He's worked in the Tenderloin for the best part of 30 years, after graduating law school and winning a grant to litigate for the Tenderloin Housing Clinic. "And yet, when you say, 'Gee, why is it that you don't see this in Noe Valley?', they just shrug," he adds, pointing out San Francisco's other, upscale neighbourhoods don't suffer the same problems. "It's just wrong."

It would be hard to argue city politicians aren't aware of what goes on in the Tenderloin – after all, everyone else is. And the theory it's allowed to happen because its residents are low-income, and it can't be sold off to developers, is equally tough to disprove. But the beginnings of hipsterfication here might offer a new option. The area's reputation means it's the city's last bastion of affordable rents. This is what is attracting cool new businesses that could put it back on the map. And making the Tenderloin visible, instead of San Francisco's dirty secret, could obligate officials to clean it up.

POWER TO THE PEOPLE

"If the city isn't going to do anything about the neighbourhood, we will," declares Stephen Yang. The young hotel developer is dressed according to San Francisco tradition: jeans, t-shirt, trainers. Yang recently transformed a fading old Tenderloin hotel into its first boutique one, The Tilden. We're sitting in the bright, leafy lobby, which he hopes will bring new energy to the area: laptop-tapping freelancers are encouraged to park up on the velvet sofas and use the free wifi, as well as the adjoining café's Equator Coffee (a local indie roaster, natch).

The Tilden's cocktail lounge The Douglas Room is the work of Mo Hodges, a local bar owner, complete with cap, glasses, nose-ring and beard. He also says it's new businesses that will clean up the mess the city has been sweeping under the carpet. "We are taking back a part of San Francisco that our city's leaders have neglected for too long," he tells me. "We don't see it as gentrification; we look at it as restoring it to its former glory."

The lobby at the Tilden Hotel, San Francisco

The lobby at the Tilden Hotel

And there are former glories. The Tenderloin is a big part of San Francisco's story, but it's not one any tourist board promotes. So, in 2013, Shaw founded the plucky little Tenderloin Museum, right in the heart of the neighbourhood, to tell it instead. Here, I learn the Tenderloin has been pissing city officials off for more than 100 years.

In the early 1900s, its dance halls, bars and brothels bred moral panic; come 1920, this sordid rep saw it flourish under Prohibition, packed with illegal speakeasies and gambling dens. In the 1930s, Sally Rand performed naked fan dances at the Music Box theatre; in the 1940s, Black Hawk nightclub was a world-class jazz hub played by everyone from Miles Davis to Thelonious Monk. In 1969, the first porn film legally shown in the US premiered at the Screening Room on Jones Street, reinventing the Tenderloin as a premier den for blue movies, lit up by adult cinemas, strip clubs and massage parlours.

Much of the city's proudly progressive politics can be traced here, too. Its apartment blocks, rather than family homes, drew single working females who helped California women win the right to vote. In 1963, the Glide Memorial Church appointed a young African-American minister, Cecil Williams, who made a point of welcoming hippies, homosexuals, prostitutes and drug addicts. (Glide remains a major provider of social services today.) And when, in 1966, a group of transgender women rioted against mistreatment by police on Turk Street, it was the nation's first gay militant uprising.

IT TAKES A VILLAGE

There is, Shaw of the Tenderloin Housing Clinic says, a silver lining to decades of neglect. "We don't have chains because we didn't get that investment," he explains. This makes the Tenderloin ripe for independent business. And contrary to the rest of San Francisco, where wealthy residents tend to commute out to Silicon Valley or the Downtown Financial District, people both live and work here. "It's like a little village," says Shaw, "you always run into people you know." The Tilden's Yang agrees; the business community, he says, pulls together: "We send each other customers, help each other out."

Other businesses go further. A few blocks up from the Tenderloin, before you reach affluent Nob Hill, you'll find disputed territory: some call it 'Tenderloin Heights', others 'Lower Nob Hill'. Truly, it's a mix of the two; grit and grime, design shops, expensive restaurants. The unfortunate compromise: 'Tendernob'.

the tenderloin is like a little village. You always run into people you know, and the community pulls together

Here, fashion boutique Hero Shop occasionally donates a portion of its profits to specific local services – like Raphael House, which helps low-income families with housing, education and job skills. Its owner, former US Vogue staffer Emily Holt, tells me: "If you're going to have a business here, you have to love and believe in it. It's important not to exist in a bubble."

Gallery owner Jessica Silverman agrees: "What we do is so different from the lives of people around us." In acknowledgement of that, she feels a responsibility to help. A member of the Tenderloin Museum board, Silverman praises the way the organisation's screenings, talks and tours coax bubble-dwellers to look, learn and engage.

Getting there

United flies from Gatwick to San Francisco from £703 return. united.com

She and Holt also plan to put on tours of the neighbourhood, bouncing guests from cool new venue to cool new venue. She admits to being shocked by the scale of the Tenderloin's problems when she opened her gallery in 2013. "I went into it a little bit blindly," she says. "But there is promise in persisting."

And there is promise in the Tenderloin's new formula for reinvigoration: one where no one, besides drug dealers, loses out. Maybe San Francisco's dirty secret will someday be its shining achievement.