(7 of 14)
Abbott spoke to Zients the next day, Sunday, Oct. 20, and flew to Washington on Oct. 21. That day, Obama offered what the New York Times called "an impassioned defense of the Affordable Care Act" in a Rose Garden statement, "acknowledging the technical failures of the HealthCare.gov website but providing little new information about the problems with the online portal or the efforts by government contractors to fix it."
Nor did the President volunteer that he had recruited a team whose first job was to decide whether to kill the website and start over.
"The first red flag you look for," says Abbott, "is whether there is a willingness by the people there to have outside help. If not, then I'd say it's simpler to write it new than to understand the code base as it is if the people who wrote it are not cooperating. But they were eager to cooperate."
"The second thing, of course, was, What were the tech problems? Were they beyond repair? Nothing I saw was beyond repair. Yes, it was messed up. Software wasn't built to talk to other software, stuff like that. A lot of that," Abbott continues, "was because they had made the most basic mistake you can ever make. The government is not used to shipping products to consumers. You never open a service like this to everyone at once. You open it in small concentric circles and expand"--such as one state first, then a few more--"so you can watch it, fix it and scale it."
What Abbott could not find, however, was leadership. He says that to this day he cannot figure out who was supposed to have been in charge of the HealthCare.gov launch. Instead he saw multiple contractors bickering with one another and no one taking ownership for anything. Someone would have to be put in charge, he told Zients. Beyond that, Abbott recalls, "there was a total lack of urgency" despite the fact that the website was becoming a national joke and crippling the Obama presidency.
But by then, Dickerson--the Google reliability guru and Burt's mentor--had arrived. "I knew Mikey by reputation," Abbott recalls. "He was a natural fit to lead this team."
Looking over the dashboard that Park, Burt and the others had rigged up the prior Friday night, Abbott and the group discovered what they thought was the lowest-hanging fruit--a quick fix to an obvious mistake that could improve things immediately. HealthCare.gov had been constructed so that every time a user had to get information from the website's vast database, the website had to make what's called a query into that database. Well-constructed, high-volume sites, especially e-commerce sites, will instead store or assemble the most frequently accessed information in a layer above the entire database, called a cache. That way, the query to it can be faster and not tie up connections to the overall database. Not doing that created a huge, unnecessary bottleneck, the equivalent of slowing down traffic on an on-ramp to an otherwise empty highway.
The team began almost immediately to cache the data. The result was encouraging: the site's overall response time--the time it took a page to load--dropped on the evening of Oct. 22 from eight seconds to two. That was still terrible, of course, but it represented such an improvement that it cheered the engineers. They could see that HealthCare.gov could be saved instead of scrapped.