I came across an interesting observation in the course of my duties this past week. A black hole. No, not one of the rapacious, no-matter-can-escape type of black hole theorized in the scientific community -A routing black hole.
I had this IPSec VPN tunnel to our DR site that has been operational for more than I would like to recall. However, within the last week we had to shut down our core router for the installation of an EHWIC Card. That was a success. Upon boot-up, i received a notification that our beloved GRE over IPSec tunnel would not come up. We rely on this tunnel for data replication, exchange DAG, and so on. To say that i was disconcerted would be an understatement!
And while technically this solution is not supported by Cisco, it had been working for 2+ years, so the thought of migrating it because it was not working any more was unwelcome.
No matter what I tried to bring up the tunnel, nothing helped.
I cleared the IPSec tunnel ad infinitum, modified routing tables, cleared the bridge configuration, no go...
Scrutinized crypto maps, ACLs, the works
Even though i knew no configurations had changed recently on any of the core devices, my feeling of desperation almost led me to do the unthinkable -reloading a production firewall.
I continued to comb the mountain of evidence left behind by debug logs. Nothing. Everything appeared to be working as it should. But mysteriously the tunnel wouldn't come up.
I theorized as to the cause on my way home; while alone in bed; in the shower; plowed through my catalog of old Cisco certification books that have now gathered a fine layer of dust in my book shelf.
Nothing doing.
Its one of those situations that make you question your understanding,
Even as I broke for the weekend, this inexplicable occurrence remained at the back of my mind like the proverbial shadow in the dark. Movies distracted me for a while, but no sooner was i done than the problem returned to the foreground.
Today I decided to give it another shot.
While poring through the vast library of the internet, I came across someone who had a somewhat similar problem. I had googled before, but different keywords do really send someone in different directions. The keywords i had used previously had obviously failed to unlock the deadbolt on the door. Today was different.
This particular guy intimated that the clear conn command had brought up his previously dead GRE tunnels.
Without any hesitation, and goaded by desperation, I ran the command on both ASAs.
The GRE tunnels came up!
Some times the biggest of problems have the simplest solutions. But this also appears to be a bug, as referenced in the Cisco bug ID CSCse36327.
While technically this wasn't a black hole, I named it so because traffic was getting lost without a trace.
No comments:
Post a Comment