Saints conceded an 80-metre interception try with just eight minutes on the clock. Romain Martial timed his run superbly to prevent what would have been a score for the visitors in the corner to bring the Stade Pierre Antoine crowd to its feet.
But instead of capitulating, Saints battled their way back into the contest against the French champions, Hartley grabbing an opportunistic try from close range and Stephen Myler kicking the remaining eight points.
However while the visitors came within a few metres of scoring what would have been the winning try in the final few minutes the Castres defence held firm to secure the result.
Despite that early score being the difference at the final whistle, Hartley was in a contemplative mood when discussing the incident.
"It's one of those things, isn't it?" he told Sky Sports. "It was only a couple of minutes into the game. We were in a good position on the field, were attacking and our shape was good. It was one of those passes that went astray and if it had been a yard deeper we'd have been in at the corner. We spoke under the sticks about getting back into the game; we clawed our way back and could have won the game."
Despite Castres coming into the game with an unbeaten home record - Toulon, Racing Metro and Stade Francais all numbering among their victims at the Stade Pierre Antoine - Hartley insisted that the visitors were in a confident mood going into the contest.
"Last week we came away with a draw, and we came here with confidence that we could go away with a win," he commented. "But they are French champions and while we thought we could win that game we'll take the point. You take every point you can get in the Heineken Cup, especially on the road, and we'll welcome them back to our place in a few months' time for the return fixture.
"I felt that we had some good moments and momentum and at times in the driving maul we got the upper hand, but on the flip side they were dangerous as well. Their off-loading game meant they got the ball away quickly and they could get some flow into their game.
"At the end of the day this is Castres. They've got a great home record, are a proud team and aren't the champions for nothing."
Saints have their own home record to defend next Sunday as the Ospreys come to town in the second pool match of the 2013/14 Heineken Cup. The team needs your support, so make sure you book your tickets throughout the weekend and get behind the players as they take on an Ospreys squad which includes five British Lions and a host of Welsh internationals.
All you have to do is CLICK HERE and follow the straightforward instructions!