For a man whose latest book outlines his own extended 'Dark night of the soul,' the popular author, spiritual director and retreat leader Mark Yaconelli is in infectiously good spirits.

In his own new work Being Disciples, Rowan Williams describes holiness as the ability simply to make others feel better. In which case Yaconelli – a writer whose anecdotal style differs from that of Williams but whose themes of individual human dignity occasionally overlap – could be described as a holy man.

For both in his latest book Disappointment, Doubt and other Spiritual Gifts, and in person, Yaconelli has a talent for turning stories of grief, disappointment and pain into tales of grace and hope.

We meet at the Greenbelt festival in Northamptonshire, where Yaconelli is a star turn and whose talk based on the themes of his book was standing room only with an over-spill crowd outside the tent.

Those themes are described in the book, published by SPCK, as "middle sufferings of life – burnout, shame, vocational failure – the frustration, self-doubt, and emptiness that can slowly corrupt our ability to access our deeper spiritual capacities".

Yet from each of them there emerges tear-inducing stories of redemption, from his own road trip to recover from going spiritually "blind", to the peace that surrounded the death of his father.

We start by discussing the personal context of his latest book. "There is always a tension for me in my own spiritual life where I am doing everything I can to be perfect, to do things right, to be holy and I've got all these spiritual practices and diets and workouts that I'm failing at but I'm back on the horse trying it again," he says. "And so there's a part of me that's always wanted to be perfect or to be holy or to just be a very moral good person, but when I look back on my life as I'm getting older, all the insights I've really gained in my life have come out of failure and struggle and difficulty, and worse than that for me, sometimes I've noticed that God has no interest in my perfection projects, no matter how holy or spiritual they appear. And I think that's the gift of failure, frustration, loss, all these kinds of human experiences – we come in contact with a kind of grace that we don't allow ourselves."

Yet arguably, there is sometimes all too little room in certain church circles for such "human experiences". Asked if there is a problem of enforced happiness in the Church, Yaconelli concurs and goes further. "Yes. I think most churches are not safe...It's, 'If I really tell you the truth about who I am and what I'm suffering through, you'll reject me or you'll question me or you'll judge me, you'll put me on the outside, I won't be nominated for the church board or anything like that. So I have to sort of check my humanity at the door and pretend' and in fact in most churches, the outside world would understand... if we had a sign over the door saying 'Let's pretend' because it feels like that's the mode in most churches – this sort of 'How are you doing today?' 'I'm doing great.' 'Isn't it a beautiful day?' 'Yes it's wonderful' even though I'm going through a divorce or I'm suffering with mental illness."

Yaconelli describes his own experience, which came at a time when the prominent public speaker was at a peak as what he calls in the book a "professional Christian". He says: "I think the dark night was difficult for me because I was at the time up in front of people, and teaching a lot, but it was almost like I'd gone blind – like I couldn't see or locate where God was. It's like waking up in the morning and your spouse has left you a note that says 'I've gone' – you know she still exists but you don't know where she is or how to get hold of her. And at the same time I was leading a big project across the US on prayer with young people, so that was a very difficult place to be because I didn't know how to behave in a way that felt honest and I had to rely on my community who was praying with me – I had to trust them to tell me what I should be teaching and what I shouldn't."

It was then that a friend, Kirk, intervened, flew to California and took Yaconelli out on a trip that brought the author back to his senses. "I was really burned out and empty and alienated from myself – I'd been working too much and a friend of mine sort of came to the rescue and took me on a trip that sort of woke me up again and brought me back to life," Yaconelli says. The passage culminates in the pair enjoying an exceptional meal together which Yaconelli describes in the book as "communion".

The reader is left wanting similar experiences, a glimpse of heaven on earth. Can we all experience "communion" in our daily lives? Yaconelli believes we can – and do. "Communion happens all the time," he says. "Sometimes we notice it, sometimes we call it that, sometimes we don't. One of the things I do is I tell stories a lot, and stories are a kind of communion, because what neuroscientists are finding out is that when I tell a story often the same places that are being stimulated in my brain are being stimulated in yours, when I hit a place of humour, we both laugh together; a place where I'm touched, if you're really listening you're touched in the same way, so we share that human experience. In the Christian faith – people are telling stories, and as we share the experience of the story there's a kind of communion that happens, a co-mingling of spirits longing in the same direction."

This "spirits longing in the same direction" is not exclusive to believing Christians, Yaconelli passionately believes. "Not only do I think that God loves everyone, I would go beyond that to say that every single human being knows God," he says. "They may not know the name [of God], but every human being has religious experiences, moments where they feel deeply connected to their own heart, deeply connected to others, moments where they feel a deep sense of compassion. I've led retreats with secular groups who say 'We don't believe in God but we want you to lead a retreat' – often I do practices that are prayer practices – I just don't use that word. And people will come out of the silences saying 'I experienced a deep sense of love' or 'I remembered a moment where I was really touched'. Now I call that experience God. They can't use that language because of something they've suffered or because of some theological issue but yes, everyone's loved by God, everyone knows God."

Sometimes however, even Christians are all too reluctant to worship. In another powerful chapter, Yaconelli tells the story of a prayer service he worked painfully hard to create at the Southern Oregon University. Having canvassed the college and gone to great lengths to promote the service, no-one turned up, apart from the three musicians and three cooks he had hired to help. Eventually, with perseverance and patience, Yaconelli oversaw the evening service coming into fruition. "After the first night I wanted to quit. And throughout it I was continually disappointed – I was disappointed in God for not bringing people, because I was showing up. I was disappointed in the people who weren't coming – I was angry at the college kids who weren't sowing up," he says. "But over time it became humbling because I had to sit through that prayer service and sit in silence and be with God every time I led it and in that silence I began to learn how to just accept the beauty of the prayer whether people were coming in or not."

The book's final – beautiful – message of redemption is on grief, and helps to conquer our darkest fears with a description of what Yaconelli describes as the "grace" of the death of his father, the well known Christian writer and speaker, Mike Yaconelli. "I do have peace about my dad's death," he says. "I have peace about my own death, not that I want to die – I want to live as long as I possibly can – and it's not because of my beliefs, it's because my beliefs have led me to certain experiences, where in my own dad's death, there was a deep sense of peace that even other family members who are not necessarily Christians felt it...I have no idea what happens in the afterlife, but I do trust that all will be well, and I trust God and I trust my experiences of God that somehow life goes on, that we're carried, that we're held in a way beyond our deepest fears."

Moving beyond the contents of the book, Yaconelli has much to say about the state of the world, including the US. The pacifist, who describes in one chapter protesting against the disastrous 2003 invasion of Iraq, is surprisingly scathing about the legacy of Barack Obama. "Sadly, if you pay close attention, Obama's foreign policy was not that much different than that of George W Bush," he says. "We've invaded eight Muslim countries. I went to Calais to visit refugees there – most of those refugees are a result of American military intervention. Obama seems like a very nice guy – he'd be great to have as a friend or a neighbour, he seems like a kind person – but the violence has continued, and the tie in to Wall Street and money has continued; no-one was prosecuted for all the financial miss-hap that caused world-wide suffering – no-one was prosecuted. So it's very difficult – you have to look past the facade because people like Obama – I like Obama – but when you see what his actions have been, the amount of people who have died world-wide...it's deeply troubling and I feel deeply ashamed."

Turning to the UK, Yaconelli, who has just finished a six month residency at the Diocese of St Asaph in North Wales, has a shocking message about the state of the Church here, but one which has a characteristically happy ending. "The Bishop asked me to speak at the Cathedral to all of the priests, all of the hierarchy in that diocese in that diocese and the people who had gathered – 500 people – and what I felt like I needed to say was that the Church has died in the UK; it's not dying – it died – and it's not coming back," he says. "The Church that people have known in their childhoods or that they knew when they were growing up – that Church has gone, and it's not coming back. And it's not their fault: God brings things to life, and then God takes things back into the ground, and nature tells us that all the time, and so the formations and the structures and systems of what the Church has been in the UK have died. There is a shell – there's still something there but it's [essentially] gone."

However, he goes on: "And the point that I believe the UK is at, is that this is an age of creativity. This is a time for new experiments, new dreams, it's a playful time, to try to figure out what helps God come alive in people today, and it's going to mean a lot of experiments, and the new Church is going to show up way past my life time, but what the future generations need us to do right now, is to trust God and to take risks and to be playful and to try a lot of experiments. And here at Greenbelt that's what's happening – there are all kinds of experiments: some are going to last a year, some are going to last three years and fail, but things will be learned in a new form that's coming and we need to be brave and courageous and radical about that."

On this as on everything with Yaconelli, there is a silver lining. "I'm very hopeful," he says. "God is not any less available now than God was 2,000 years ago. God is still present. And people are being touched and lives are being changed but what's sad is that we tell ourselves a story that things are bad and terrible...Death doesn't have to be a time that's absent from God – it can actually be as many people will tell you, sometimes when someone's dying, it's a thin place where God is present – we're in that time now, and God's evident when people have eyes to see and ears to hear and a heart willing to take chances."