The curious case of the black hole

I came across an interesting observation in the course of my duties this past week. A black hole. No, not one of the rapacious, no-matter-can-escape type of black hole theorized in the scientific community -A routing black hole.

I had this IPSec VPN tunnel to our DR site that has been operational for more than I would like to recall. However, within the last week we had to shut down our core router for the installation of an EHWIC Card. That was a success. Upon boot-up, i received a notification that our beloved GRE over IPSec tunnel would not come up. We rely on this tunnel for data replication, exchange DAG, and so on. To say that i was disconcerted would be an understatement! 

And while technically this solution is not supported by Cisco, it had been working for 2+ years, so the thought of migrating it because it was not working any more was unwelcome.

No matter what I tried to bring up the tunnel, nothing helped.

I cleared the IPSec tunnel ad infinitum, modified routing tables, cleared the bridge configuration, no go...

Scrutinized crypto maps, ACLs, the works

Even though i knew no configurations had changed recently on any of the core devices, my feeling of desperation almost led me to do the unthinkable -reloading a production firewall.

I continued to comb the mountain of evidence left behind by debug logs. Nothing. Everything appeared to be working as it should. But mysteriously the tunnel wouldn't come up. 

I theorized as to the cause on my way home; while alone in bed; in the shower; plowed through my catalog of old Cisco certification books that have now gathered a fine layer of dust in my book shelf.

Nothing doing.

Its one of those situations that make you question your understanding,

Even as I broke for the weekend, this inexplicable occurrence remained at the back of my mind like the proverbial shadow in the dark. Movies distracted me for a while, but no sooner was i done than the problem returned to the foreground.

Today I decided to give it another shot.

While poring through the vast library of the internet, I came across someone who had a somewhat similar problem. I had googled before, but different keywords do really send someone in different directions. The keywords i had used previously had obviously failed to unlock the deadbolt on the door. Today was different.

This particular guy intimated that the clear conn command had brought up his previously dead GRE tunnels. 

Without any hesitation, and goaded by desperation, I ran the command on both ASAs. 

The GRE tunnels came up!

Some times the biggest of problems have the simplest solutions. But this also appears to be a bug, as referenced in the Cisco bug ID CSCse36327.

While technically this wasn't a black hole, I named it so because traffic was getting lost without a trace.

We have seven months

New year's day 2017. The arrow of time surely surges only forward, and quickly. And for our beloved country, this year marks yet another moment of uncertainty with a General election slated for August. For those that may recall, historically at times August hasn't been a particularly pleasant month for Kenya. Add to that, as Ndii put it, in previous elections where there has been an incumbent gunning for a second term there has been at least some violence; 1992, 1997, 2007.  Transitions have been smooth for the most part.

But all is not lost.

Recent changes to the electoral laws have increased bile in some quarters. But this is no surprise since these same quarters would only quit their vituperation if they were declared the victors in the election. But their case is unsound. Put simply, the amendments to the law allow for a manual system of counting, tallying and submission of electoral results should the electronic system fail. We have a strong need for this because, for one, a lot of areas are not covered by the cellular communications network, infrastructure is unrelaible, and lastly systems do fail.

Anything short of a fall back system is a wild stab in the dark. What would then happen should the system fail? I understand the opposition's angst regarding possible ballot stuffing etc. But such irregularities can be overcome only if the arbiter is credible and non partisan. But having only one system does not automatically make the election credible. What we need is an electoral umpire that's beyond reproach, something we haven't had in Kenya in a long time.

And that should be the opposition's (and our) main concern.

Calls for mass action are really ways to beat drums of war, and psychological priming of the masses to an eventual demonstration when the election is called. 

We need to find a common ground before the fat lady sings. We have had an eterniny, since 2008, to mend fences. But we have continued, in various ways, to tear them down.

The fact that we are still in such an uneasy state this close to the election is enough to give one flatulence.