In a sun-drenched courtyard under a deep azure sky, a breeze stirs fuchsia-pink bougainvillea blossoms that grow artfully around a stone bench. More fluoro-hued blooms peep out from a few, sparing flower beds, their stems swaying delicately in the wind, and in the distance I can see rocky hilltops cut in CGI-sharp relief. It's almost completely silent, and even birdsong is fleeting. I am completely alone.

A door clangs shut, shattering my reverie. I begin to register other details: the surrounding walls are thick and twice my height, topped with glittering shards of broken glass; at every doorway swings a rusted metal gate; and over the entrance to the courtyard is a stone platform, complete with gun mounts. Carved into the stone are the words 'Trai Phu-Hai'.

Because despite this spellbinding beauty, I'm standing in an abandoned prison on Con Son, the largest island in the Con Dao archipelago, 100km south of mainland Vietnam and an hour's flight from the Vespa-packed streets of Ho Chi Minh City. Here, in this perfect tropical paradise, is where first the French and then the Vietnamese government improbably chose to imprison and torture thousands of political dissidents for more than a century.

View on Instagram

Phu Hai, built by the French colonial government in 1862, was just the first prison here. As resistance to the French (who colonised Vietnam from the mid-1800s until 1954) grew, so did the number of prisons, their population averaging 2,000. All the prisoners at Phu Hai, up to 200 per room, were kept naked, chained to a metal bar. 'Solitary' rooms, for those considered to be the most dangerous or troublesome, held up to 63 prisoners, the space too tight for them to do anything other than stand.

When the French withdrew, the prisons in Con Dao came under the control of the US-supported Vietnamese government, which used them to hold members of the insurgent Vietnamese Communist Party and other political protesters. By the 1970s, the island's inmate population rose to more than 10,000.

The archipelago was liberated in May 1975. Prison guards, administrators and their families locked the prison buildings and fled; within hours, the prisoners freed themselves. The 11 prisons fell into disrepair, and the government gradually began to think about transforming the island into the next must-visit holiday destination.

Which is how, 43 years later, I come to be standing in this spot, taking in one of Con Dao's 'tourist attractions'. I wander into one of the detention rooms, a dust-coated concrete cell where you can still see the metal bar to which prisoners were shackled. Emaciated stone mannequins illustrate all too well how it would have worked. Their expressions, mouths stretched open in pleas that will never be heard, are haunting.

Here, in this perfect tropical paradise, is where thousands of political dissidents were once imprisoned

These immobile faces stay with me as I continue my explorations. I no longer feel alone; there are ghosts with me, my imagination spinning stories in the whisper-like rustle of dead leaves blowing across the floor, and in the chill that raises the hairs on the back of my neck when I step out of the sun's reassuring warmth.

And then, suddenly, my solitude ends as a crowd of Vietnamese tourists shuffles around with me. The idyllic bench spot that captured my attention at the entrance becomes the setting for an Instagram-worthy photo, full of smiles and laughter. The prison's eerie spell – whether real or imagined – is broken.

My guide, Trang, who has been discreetly waiting for me as I roam in pensive silence, explains that visiting Con Dao and its prisons is a ritual; the survivors and their families coming back, sometimes on a yearly basis, to pay their respects.

We leave Phu Hai and cycle down deserted streets to Con Dao prison. This is Con Son island's most well-known institution, where inmates were kept in horrific tiger cages and solariums (so-called because they had no shelter from the blistering sun). We bump into the same group of tourists, along with several others – all of them Vietnamese.

This is one of the things that strikes me most during my trip: most of those who visit the Con Dao archipelago are Vietnamese tourists, who stay in guest houses dotted around Con Son. As well as the prisons, they come to visit the cemetery, where those who didn't survive the conditions were buried.

Around 20% of the Vietnamese tourists come to visit the shrine of Vo Thi Sau, a girl who joined the resistance against the French aged 14 and was sent to Con Dao to be executed. A fire burns for her day and night, with visitors coming in their hundreds – especially at midnight – to pay their respects. We drive past the shrine as we head out of town, attended by pilgrims bearing gifts.

And then, a few miles down the road, we drive past a modest entrance: Six Senses Con Dao, the first mark of the island's step-change. The famous luxury boutique hotel collection is the first proper resort to arrive on the island, bringing with it its trademark sustainable ethos and plenty of wealthy patrons – Brad Pitt and Angelina Jolie once stayed here together. Until now, as well as its wealthier visitors, Con Dao has also been the preserve of keen divers; its waters are incredible for diving, with dugong, turtles, dolphins and more than 200 types of fish.

But this gem won't remain under the radar for long. Con Son commands a brooding beauty: despite the sunshine of my morning's adventures, throughout the rest of my stay the sky remains moody. The sea is a sulky grey-blue and the green of the jungle is impenetrable, but that serves as the perfect foil to the bright pink bougainvillea, which shows its face around each and every turn.

This lush, untamed landscape reminds me of a Vietnamese version of Daphne Du Maurier's imagined Cornish estate, Manderley. The vegetation is completely different to anything I've seen before – tropical paradise it may be, but the Con Dao archipelago feels worlds away from your stereotypical desert island.

With such natural beauty ready for the taking, it's hardly surprising hoteliers are eyeing the island for their next outposts, and some are already in the process of being built. I'm staying at one of the newest additions, the Poulo Condor hotel on Con Son's eastern edge, a self-appointed 'affordable luxury' resort and spa.

Poulo Condor has an irresistible selling point: it sits within the Con Dao Nature Reserve, which means that beyond its paths lies pure, untouchable rainforest. In the middle of the hotel is a lake where you can go kayaking; in front of my room is a vast pond filled with pink lotus flowers; and there's a pristine beach, thanks to a daily litter-picking programme that guests can get involved with on Saturdays. I while away several hours making lazy strokes through the pool, its grey-green tiles reflecting the moodiness of the skies and sea.

This access to the rainforest also means islands are primed for hiking, and I spend a morning in the jungle. Vân, my guide, points out a giant black squirrel; bow-fingered geckos scurrying around on rocks; and curious (and probably hungry) monkeys that keep us company as we scramble up the hill. She also retrieves me, laughing, when I trip and fall headlong into the undergrowth.

As we walk, I can hear a tinny voice being broadcast from speakers below, on the road. Vân tells me there are speakers all over the island, announcing hourly news and weather – and I'd wager, propaganda.

Because, despite the wilderness that I admire so much, there are hints of a regime. By the roadside, I notice soldiers sleeping in hammocks. Cristy Gonzalez, Poulo Condor's general manager, tells me that 7,000 people live on the island; 3,500 of that are military, taking care of the archipelago in case "someone comes to take it".

But for the most part, Con Dao is a deserted idyll. I spend another morning cycling around with Trang, nodding to local cyclists wearing the classic circular pointed hat. We climb up to the Buddhist temple and drink refreshing iced water filled with chia, the seeds plump and swollen like frogspawn; we visit the market in the centre of town, where I watch women crouching to prepare fish and fruit; and we cycle down tree-lined boulevards, past the verandas of crumbling French villas. There are subtle glimpses of the past in the stone plaques and abandoned buildings dotted around the place, but it feels like the island itself is trying to forget, laying out treasures for me to admire and seducing me with dramatic romance.

In the evenings, the sleepy town stays, well, sleepy. Most of the restaurants are geared towards Vietnamese visitors, but this means they serve delicious, authentic (and cheap) local food. Cristy and her colleague Tommy take me into town and we drink strong Vietnamese-style iced coffee at the dive shop, which also doubles as one of the island's most popular restaurants, serving burgers to hungry backpackers.

Need to Know

InsideAsia offers seven nights in Vietnam from £1,295pp, excluding international flights and including three nights at Fusion Suits in Ho Chi Minh City; four nights at Poulo Condor Con Dao; guided day tours of Ho Chi Minh City and the Mekong Delta; airport transfers and return flights from Ho Chi Minh City to Con Dao. Vietnam Airlines flies from London to Ho Chi Minh City from £655 return.

It would be hard to find a starker contrast to the noise of Ho Chi Minh City, famed for its pulsating nightlife and frenetic pace, where you can eat street food in the early hours of the morning, drink cocktails in rooftop bars and take tours on the back of a scooter. But that, too, has its charms, and it's not hard to see why visiting the two together is a perfect pairing, not least because of their proximity to one another. The juxtaposition within Vietnamese culture should be jarring, but it's spellbinding.

I come away from Vietnam feeling completely bewitched. Con Dao and its ghosts don't haunt me anymore, but that enchanting atmosphere is something that I promise myself never to forget.