OVH and the unfortunate “human error” which brought down thousands of French sites

Many sites were inaccessible for more than an hour this Wednesday, October 13, 2021. The French host OVH, which was preparing its IPO the same week, confirmed to be involved and spoke of a “human error”. It could be a simple copy and paste badly done. Here is the whole story.

What do the site of the Centre Pompidou, Strasbourg airport, the media Clubic and Stop on Images and data.gouv.fr? They were all completely inaccessible on October 13, 2021 around 9:30 a.m., Numerama’s editorial staff confirmed.

At issue: the French host OVH, which confirmed a problem at its level. It would be a ” human error », According to its co-founder. It was fixed in an hour.

On the site DownDetector, where Internet users can report breakdowns on the sites they visit, 14,000 reports were observed in less than 30 minutes, which is very significant. It was not only France that was affected: OVH also has customers abroad (” 1.6 million customers in more than 140 countries ”), In Canada, Spain or Turkey for example.

Outages reported on OVH on October 13, 2021 // Source: downdetector

On Twitter, several internet users made the same observation.

What happened with OVH?

The routine maintenance of a router, scheduled for October 13 between 9 a.m. and 10:30 a.m., seems to have gone wrong. This is what confirmed Octave Klaba, co-founder of OVH, on Twitter: “ Following a human error during the reconfiguration of the network (…) we have a problem on the whole backbone », He published.

« In recent days, the intensity of DDoS attacks has increased significantly. We have decided to increase our DDoS processing capacity by adding new infrastructures in our DC VH (US-EST). A bad configuration of the router caused the failure of the network “, He then clarified.

In a statement released at 11:35 am, the company confirmed this, specifying that “ OVHcloud teams quickly intervened to isolate the equipment at 10:15 am. Services have been restored since this intervention. We are currently running a verification campaign with our clients to confirm the restoration of all their services. »

Can we speak of an OVH “failure”?

According to the explanations provided by the founder of OVH Octave Klaba, the problem with the French host would have come from a concern of network configuration. While maintaining a data center in the United States, a configuration error would have prevented the routing of IP addresses, thus preventing access to a large number of sites.

To put it simply, connections made over the old IPV4 protocol seem to get lost in the wild, while those using IPV6 don’t seem to have a problem.

In a tweet (since deleted), Octave Klaba hinted that it could be a stupid story of … failed copy and paste, which could happen to anyone: in a piece of the code, the number ” 4 ”which should have been appended to“ IPV ”has been moved to the next line, making the term ipv4 incomprehensible.

Source : Twitter/olesovhcom

The maintenance of the server responsible for this bug was a priori done to deal with DDoS attacks including ” intensity […] has increased a lot […] the last days “. It is undoubtedly by putting the server back on its feet with a bad network configuration that this bug intervened.

Can we therefore speak of “failure” properly speakingr? Well yes, anyway, since many sites are inaccessible by most Internet users, whatever we say. This is due to the still massive use of the IPV4 protocol, which is outdated today. But the data centers themselves are not down and customer data is not at risk a priori.

Servers were available, but not found by much of the global web. In short, it’s a bit like writing a postcard to ” the green house 300 m from the church in my village “. No matter how much this house exists, it will be difficult for the postman to find it because of the use of a somewhat outdated addressing method.

IPV4 / IPV6: why does it make a difference?

The Internet is a vast computer network to which millions and millions of devices are connected – servers that host websites for example, but also computers, smartphones and many connected objects. However, in order to differentiate all these terminals from one another, each of them must be assigned a code. You have to imagine it as a kind of license plate.

We can also see this code as a kind of call number: by knowing this code (called an IP address), we can thus be directed to the right destination, the right recipient. When the current system, called IPv4, which is still widespread on the net, was invented, its structure made it possible to create the equivalent of 4.3 billion license plates.

An IPv4 address is a series of four numbers that can range from 0 to 255 and separated by periods (for example 172.16.254.1). In 1983, the year IPv4 was introduced, 4.3 billion plates were assumed to be more than enough. Then the Internet became more democratic, connected devices flourished and the Internet of Things appeared. And today, we are reaching the limits of IPv4.

To put it quickly, almost all available IPv4 addresses have been assigned. The good news is that a new version of IP addressing has been invented: IPv6. This makes it possible to obtain 340 sextillions of addresses, that is to say 340,000,000,000,000,000,000,000,000,000,000,000,000,000 addresses (yes, there are indeed 37 zeros). Beside, the number of stars in the observable universe is almost laughable.

And what about IPv4, with its 4,300,000,000 assignable addresses? The structure of an IPv6 address is very different. It is made up of eight blocks, each of which accommodates four hexadecimal characters (letters from A to F and numbers from 0 to 9), with punctuation to separate them. Thus, if an IPv4 address is written 172.16.254.1, an IPv6 is written 5800: 10C3: E3C3: F1AA: 48E3: D923: D494: AAFF for example. With so many IPv6 addresses, we have plenty to see.

In principle, everything should roll with IPv6. The point, however, is that the transition from the old protocol to the new one is not going as fast as the depletion of IPv4 stocks. A seesaw is certainly engaged, but the sector is moving forward in dispersed order on this subject. Some are early or have completed their update, while others have a long way to go. In short, the net is on the verge of saturation.

By the way, what is OVH?

OVH, which is now called OVHcloud, is a French company founded in 1999 by Octave Klaba. In the space of 20 years, it has become one of the heavyweights in the field of website and file hosting. If it is still of modest size compared to American behemoths like Amazon, Google or Microsoft, it is the leader in France and in Europe.

To underline the evolution of its activities (but also the evolution of the practices of Internet users themselves) and to make its core business much more obvious to everyone, the company changed its name in 2019. Indeed, cloud computing and remote hosting constitute, according to data from October 2019, more than 70% of its turnover.

In recent years, OVH has positioned itself as a company concerned with the challenges of digital sovereignty. It could thus recover within a few years the Health Data Hub project, a huge platform relating to the health data of French people. It has also approached a German competitor and obtained decisive certifications in this context.

However, this story is marred by some notable technical incidents, as in 2017 with a power supply problem. But it was in 2021 that the group experienced its biggest disaster, with a fire that devastated a large part of one of its data centers in Strasbourg. The restart took a long time and it was found that sometimes data did not survive the fire.

Which sites were affected by the problem with OVH?

Thousands of sites were affected by the October 13 blackout. The list below is therefore not exhaustive, but it gives a fairly clear overview of how major this incident is on the French web.

  • arretsurimages.net
  • clubic.com
  • data.gouv.fr
  • demarches-simplifiees.fr
  • interflora.fr
  • humanoid.fr (this is the site of the Numerama group 😭)
  • lucca.fr
  • macg.co
  • strasbourg.aeroport.fr

OVH’s “impact-free” maintenance already hijacked

Paradoxically, the tweet announcing the maintenance ensured that there was ” no impact expected “, because the ” devices will be isolated before the change “. Obviously, this did not fail to react Internet users, who used their mocking memes to hijack the company’s communication.

Source : Twitter/DeadkatFS
Source : Twitter/suprnurd

An embarrassing timing: OVH must go public in 2 days

«OVHcloud: the IPO of one of the greatest successes of French Tech becomes clearer», announced OVH itself, in a sponsored article on the Frenchweb site, published … on October 13, 2021.

The timing couldn’t be more troublesome, as thousands of sites are down because of a problem… within OVH. The company has been preparing for its IPO for months: it must take place this week, is one of the most important in the world of “French tech”. The company should be valued at around 3.5 billion euros.

However, the magnitude of the problem should be put into perspective: an error like this can happen, and OVH reacted very quickly. The fact that it is a “quack” rather than a more serious problem is even reassuring on the infrastructures and the functioning of the company and its services.

Leave a Comment