Archives

Plenary session
23 May 2016
4 p.m.

CHAIR: Good afternoon, ladies and gentlemen, welcome to the afternoon Plenary slot of RIPE 72 on day 1. If you could come in and take your seats so we can continue.

Before we start this afternoon, just a quick reminder that all of the Plenary talks and indeed a growing selection of the Working Group talks can be rated by the audience. This is really useful for us because it gives us information on what kind of material you are interested in. And also, you can win a prize. Additionally, you can nominate members for the PC, this can be done, an e‑mail has been sent out. If you wish to be nominated please send an e‑mail with a picture and a little bit about yourself to pc [at] ripe [dot] net. This afternoon's session is chaired by myself Brian, and Filiz. It is the action‑packed session, we are naughty people, we have zombies, we have hijacking, we are flying Liam Neeson in specially just to say a few words on the mike, but we have a lot of speakers, with particular sets of skills, and they will be telling you all sorts of interesting things this afternoon.

So, we're going to start with Paul Ebersman, he tells me he is going to be saying all sorts of things to be getting you excited with what's so hard about DNSSEC.

PAUL EBERSMAN: Thank you very much, before I dive into the talk I'd like to do a quick survey. How many of you are currently running a DNSSEC validating resolver? Outstanding. And how many of you are signing your own zones? Also, good. So, there's probably already a bit of pain and experience in the room.

I get to talk to a lot of people about DNSSEC and since my day job, we rolled it out starting in 2011, people are usually curious about, was it hard? How did you do it? Did your management actually buy off on it? How do you survive? And so, I decided to talk about where we are now and in some ways I think the pleasant news is that if you make good choices and test things upfront, it is not nearly the landscape of pain that it was five years ago.

So, why would one do DNSSEC? What are some of the sort of drivers, the business reasons that you need? Certainly, for Comcast one of the big divers initially was cache poisoning. The Cominskey attacks happened in 2008 and we all looked up and said how do we protect our users? And there were a lot of games that we played with, timers and randomness and entropy and other things to give us time to take a deep breath but the reality was DNSSEC was probably the most useful thing we could do at that time. One the other things that DNSSEC does is, it at least alerts you to when someone is doing lying, when I say lying it could be anything from actual malicious intent to try and hijack your domain to some of the things like RPZ, response policy zones, DNS filtering, NX domain rewrites, anything where there is more than one possible answer for DNS. One of the other big things we are pushing is DNSSEC is not just about protecting the integrity of the DNS answers you give and in other people's caches. It also does give you at least the same level of trust that you would get to the DNS itself and so it enable things like DANE and various other forms of PKIs, so we have something that is a usable already implemented distributable database that we can use.

So, when you go off and you start out with, I want to do DNSSEC, there are certainly enough people that will tell you every reason in the world why DNSSEC is wrong, broken, evil, hard. It's very confusing. I have to train my staff. I have to train my users. It doesn't really fix anything. It actually breaks more stuff that was working just fine for me. And there are even those who sit there and say, well, it means that I have to trust the DNS, and the ICANN routes and all the rest for all of my other stuff, and I don't really trust them, I don't believe in what they're doing.

So, my experience, yes, DNSSEC is hard. And you can make it much harder for yourself if you don't walk in with the mental attitude upfront that the only sane way to survive this experience is to automate everything. Test the automation. And make sure it works before you turn it on in production. And I'll talk more later on in the talk of exactly where in these sequences some of those breakages occur, but realistically at this point if you go with some of the more popular Open Source software, or any of the more stable commercial DNS packages, they have finally gotten to a stage where you really can roll it out and you really can keep it up and running and not have odd breakages beyond the normal DNS fun and excitement that we're used to.

So, automate is going to be the theme throughout all of this. It does really help with cache poisoning, which is one attack. It's not everything. It's not everything they can do bad with DNS. But in the Internet we sort of chew off the problem one at a time and this is certainly one good way to take one form of attack out of the equation.

We're starting to use DANE ourselves for authenticating e‑mail. I'm a big fan of that. Anybody who wants to grab a beer in the bar and start comparing certificate authority and certificate chain problems and issues and how the model is fundamentally possibly broken, I'm happy to have that discussion, but the answer is that for me, DANE is ‑‑ well not a replacement for CAs in all circumstances, it's certainly a better and cheaper answer for a lot of problems of why do we have certs and why do we use certs? As far as route and ICANN, I get an easy response for anybody. When was the last time you put an IP address into a URL or anything else instead of a DNS name? If you have given up and you are on SC hosts, congratulations, I applaud your enthusiasm and vigour, but for the rest of us we are already trusting DNS anyway to get us to the right server, making one more leap of trust and making that leap more reliable is, to me, a step up.

And surprisingly to me, we do have customers who come to us and say we are concerned about having DNSSEC available to us, and who actually say when we are validating, even when it's their domain and it breaks, they will say to us afterwards, thank you for not just band‑aiding it over; thank you for actually breaking correctly so I knew it was broken, and so that my DNS was kept secure.

So, how do you start off if you do not currently have DNSSEC? The two halves of that equation are, do you validate? Is the recursive resolver when you are looking for the answers, do you walk the chain of trust and do you fail to give answers if that chain of trust is broken demonstrably in some way? Somewhat more ambitious but useful to you to protect your own DNS date is when you actually go ahead and sign your zones, get them automatically rolling over, taking care of all the key rollovers.

So, starting with validation. Validation, it's very easy to enable. I have some examples at the end of my slide deck of some configurations of several of the Open Source packages and it really is something along the lines of you add one line to your configuration file and you start your name server. It sounds great. The fun, of course, is that as soon as you turn that on, you are basically saying, I believe that every zone owner and authoritative name server operator on the Internet is clued, competent and paying attention. It's a nice assumption. I also love fairytales when I read them to little kids, but I tend not to believe that particular story. The reality is, is that there are a lot of things, but it is certainly doable and I'd talk about what are the real business challenges and business costs. The nice thing also is that not only is it one line for all of the packages, but pretty much at this point every one of the available major Open Source packages does also support doing validation. And more or less does it correctly. A few little interpretations of RFCs that are a little different. Some perform to start with believing the child, some believe in trusting the parent, but mostly everybody supports it.

Signing: Automation is not an option. Anybody who thinks that they want to do this by hand is nuts, or does not carry a cell phone or a pager. There is just no way that you are going to do this by hand and turn it over to your first tier staff. That said, it has to be good automation. There have been a lot of failed or half‑assed attempts at doing automation over the years in DNSSEC. Quality has varied, different releases of different software have varied widely, so, shop carefully, be a very picky, very informed consumer. You want to dive deeply into exactly what they are talking about, do they handle different kinds of rollovers? Will they pre‑publish? Will they automatically resign? Will they resign on any form of insertion in a dynamic update or do you have to actually feed the resulting new zone file in the clear into some kind of signing in some form of incline or bump in the wire method? There are advantages and disadvantages to all of those. We all have different operations and different infrastructures. So, you not only need to make sure that it works well but that it actually fits your model and your first tier supports model of how things work.

Getting it working initially is not trivial. We have a development group that was running their own [auth] server because they were doing some interesting load balancing and they had a lot of time and effort invested in code, in how all this works. At some point I have a corporate directive, everything shall be DNSSEC signed and validated. I said to this group, you have your own out server, it's not one of mine, are you ready for DNSSEC? They said, what do you mean, DNSSEC? I said, are you ready to sign your records? And there's a sort of pause in the conversation and they start pulling up, oh, you mean this RFC? Yes. Oh... well, how hard can it be? We'll just add it. Yes... I will give them credit, one year later, they had not only written all of the code, they had written the automation for doing the key rollovers and they had survived a very bumpy ride through but had made it out the other end of their first key rollover including a KSK rollover, so bravo, but yes, you should be prepared for an awful lot of work, and I felt very good when they came back to me, because initially they said, how hard can it be? Afterwards the guy came to me and said, I thought you were a kind of kidding, but it took us hundreds of man hours I think by the time they are done.

So, key rollovers, that is going to be your point of pain. Again, one of the offers in the bar, if we would like, to and I know there are a few in the audience would who like to discuss how much fun is getting DNS records and DNS kids in the parents can be. Yes, KSK rollovers are of key rollovers, the not fun part.

So, let's get into so. Actual issues, things that we actually saw that sort of bit us.

So, you'll hear it a lot. If I turn on validation as an ISP, it's just going to be an nightmare, it's one more thing that my customers are going to scream about and it's going to get broken and I'm going to get the call. There is a US model, I think it was from the AOL days that said if someone picks up the phone, dials an 800 number and you have a paid employee that says 'hello', they have just cost you $25 US, and that's probably not too far off of reality. So, if one has millions or tens of millions of customers and all of them can potentially pick up a phone, that's a scary number. We also get, all the time, the, why are you blocking such and such site? And Murphy's law says it's always the gambling site, the some popular video site, and they are always convinced this is some conspiracy and we, as a provider, are obviously filtering their DNS to prevent them interest getting the content they want. Then when you try and explain, this is DNSSEC, you need to talk to the zone owner, we can't actually fix that, they go, what do you mean? It's your resolver. You have got to be able to fix it. And so, we go through that tap dance quite regularly.

But, the other side of that is, I do actually get to sleep at night. Very few things escalate to me at third tier. Not that many of them really hit our second tier these days. We are getting to the point that even in some cases our first tier folks are saying it's DNSSEC, it's validation, talk to so and so. Some numbers:

Matt Larson at Dyn and Geoff Huston of APNIC have both done studies of resolver traffic and who is doing what and their numbers match pretty well and one of the things that they both seem to agree on is that Comcast and Google both do about 10% each of the queries that are available on the public Internet. So, one out of five ISP‑based queries publicly available are handled by one of us. We validate everything. Both of us. And neither of our teams have been told 'stop that'. Neither has said, management going, why are you not getting all of your daily tasks done? Why is DNSSEC eating so much of your man hours? We do orders of hundreds of billions, give or take, queries in a given day. And we have something on the order of single digit dozens of failures that turn out to be DNSSEC a month. I don't even know how many point‑zero‑zero‑zeros that is of a percent. It's noise. We probably have almost as many simple off failures of sloppy delegations, lame delegations, all of the out name servers are dead. They didn't pay their bill so their registrar took out their records. It's not quite as much as DNSSEC failures but they are not far off of each other. So it really isn't the kind of staff drain that a lot of people I think fear. And there is a process, there's an RFC that explains something called negative trust anchors and, these days, again most of the Open Source and commercial DNS vendors will support this that if they do DNSSEC validation. What it says even though DNSSEC validation failed and I should have had give a cert fail for this. If it matches a certain white list that I have by domain name, I will give the answer instead of serve failing. You should contact the zone owner, you should verify that this is not a security attack, you should do a whole series of things and, if none of those work and it's still critical that people reach this, then you can insert this negative trust anchor, and so I don't think we have ever had, in my two and a half years there, more than two dozen labels in our NTA list at all other than a few that we have kept long term at the request of the zone owners until they can get the DNS records out the parent and design, which has happened a few times. They don't tend to stay very long and in the vast majority of cases, we don't put the NTA in. The user thanks us for explaining what the problem was. The zone owner occasionally asks us what do we mean and then we give them something like DNS viz .net output to help explain. And they fix it and life goes on.

So, what do we see that causes these? Expired signatures are an amazingly large portion of the issues. I expected those to be mostly noise and it almost always be mismatch or lack of either signatures or DS in the parent. But, just expired signatures happens more than it should and in most cases, it's either a bug this their automation or cheap or lousy or old automation.

We do occasionally see things where they have originally signed things and then they don't correctly take out either the signatures in the zone file or the DS record in the parent and there's a mismatch and it shows as bogus. Small percentage, there are a couple of registrars that have made it so easy to DNSSEC, that they just put a little check‑box: do you want your DNS secure and they go check that they didn't even though they had DNSSEC. And then, later on, they just up wipe their zone file and they are broken.

Key rollovers and particularly KSK resource probably our biggest problem.

Issues on the signing side of things. Again, it's easy to get it signed upfront or easier, but the rollovers don't work. And by far the biggest problem that we have are those mismatches. They put a DS record in originally and then they remove the signatures or they have resigned the zone and rolled the key signing key but they failed to put that fresh hash up into the parent.

Or they are unable. Again, best part of that part of the discussion, there are at least as many methods of getting those in as there are registrars and sadly I think there may be in excess of the number of registrars since some registers have more than one. It's just a problem.

So, how do we deal with it? The biggest things come up with training for your first TLs, explain what it is, give them some tools like Dig, not NS Lookup, give them DNSviz.net. Explain how it is. Similarly, get ready to train your customers. Have preset documents or web pages or Wiki pages ready that explain, that is what's happening. This is what's broken, here is who can fix it, here's how they need to fix it, here is what DNSSEC is, here is why this is not some plot. This is actually a good thing for you, the customer.

Other things that we have done in the us.gov and .mil are by far the biggest problems we have. They had mandates thou shalt do DNSSEC but they were told they must do that with zero funding for training tools, software or hardware to implement it. So, a lot of it was done very badly or in a hurry. That's one problem. They also don't use ‑‑ well WHOIS isn't wonderful for the rest of us either but it's at least better for the general WHOIS than it is for .gov and .mil. They run their own servers, so we have collected a group of contacts that we will go to so we can get to the person who owns the zone to get them to fix it quickly. And the last one is explain to your management why indeed it is important.

Negative trust anchors, read the RFC, do them correctly. And again, it should be the last resort. That is one of the things in the RFC.

And at this point, if there are questions on problems, issues, that you guys have seen, please come up to the mike.

FILIZ YILMAZ: I think the first one was Jim.

AUDIENCE SPEAKER: Jim Reid. Speaking for myself. Paul, congratulations to you and everybody else at Comcast for doing this. I think you have underplayed the siginificance of all the hard work and I think you and everybody else on your team deserves an enormous amount of credit for biting the bullet and doing that. I have got a couple of questions though so I'd like to play devil's advocate a little bit here.

I think one of the reasons why you're not having so many problems with validation fears is because there is not much signed. Have you any kind of data on how often you are getting a validation hit on your resolver as opposed to not having to do validation because the original DNSSEC isn't signed?

PAUL EBERSMAN: I don't have as much in the way of firm statistics that I'd like, and yes, signed zones are still in the sub‑v6 implementation percentages, you know, you're talking, depending on who you are and where you are, 10, 20, 30%. There are populations that are much higher. It's country ccTLDs that have done a great job of encouraging their use. .mil and .gov for all of the heavier pain and agony, they also have a higher there, in the 60 or 70 implementation range. It's shifting. But it is slow. I think things that are going to drive it are not simply cache poisoning but things like self‑signed certificates for DANE are going to be much bigger drivers for bothering to sign.

JIM REID: I think it's a great thing, Paul, you are here to say as one who is running probably one of the biggest networks in the world that the cost of doing validation isn't a significant overhead at least not at the moment for your business. If I was in the business of actually having to run a last resolver firm for a media operator, my worry would be if I turned this on the support costs are going to be so great that I'd be looking for another job at the same time my competitors have not having this disadvantage because they are not switched on validation. So maybe if I take the plunge I'm going to have to take the hit and my competitors are going to have a better perception in the public eye because they are not going to have to do validation. I think another consideration we are going to have to think about here, I had a look at probably the top ten websites: eBay, Amazon, Facebook, they are not signed. What are you going to do when they sign and they stop validating properly?

PAUL EBERSMAN: I would hope that they will do what I said and pick decent automation.

JIM REID: Yeah, but what if they don't?

PAUL EBERSMAN: We have gone through this before in a couple of cases where if it's obvious that they will be unable to do it competently in a near term, that is one of the reluctant NTA implementations that I will do. It was done at one point for one of the NAS a sites that just couldn't get their act together at the time. The wrong people were on vacation, and so they couldn't fix things in time and it was one of the Mars landing or something elsewhere there was a lot of traffic where our users were very upset. So, we have ways of working around it if we have to. But, again, I will have to give credit to the software folks, the tools are much better for doing most of the automation most of the time than they were two years ago. So, it's a lot harder to do a really lousy job than it used to be.

JIM REID: Okay, Paul, thanks very much and thanks once again for switching this on. Thank you.

JELTE JANSEN: SIDN. I have about 27 comments, I rolled it down back to two and now after Jim I think I have 54, I think. But the first one is that, as SIDN we have been promoting signing so we are up at about 45% of our zones are signed, almost 2 and a half million now, and the what we see most is that people stop signing their zone and don't tell us, so you had that as the second point, so unsigning but the DS is still up there. That's what happens by far the most.

But the other thing we saw is that because we are checking this and we are informing them that the ever rate is going down, so for our two and a half million signed zones there is about a 0.1% error rate right now and most of those are part domains where the owner just doesn't care.

PAUL EBERSMAN: And thank you for doing the cleanup.

JELTE JANSEN: And thank you for doing this because it's nice to say, well, Comcast can do it, so why the fuck can't you?

AUDIENCE SPEAKER: I actually was listening to what you said and I heard a déjà vu, there are proposals columns which have been around for a while and we could not get deployed and you guys do a great job in both deploying them, both IPv6 and DNSSEC. Thanks for doing this. I'm curious, which one was harder?

PAUL EBERSMAN: I know what John Brozozowski will say and actually I think that, in many respects, IPv6 is harder because the network sack is embedded more objects, there are more networking devices and network harbour vendors seem to be a lot more responsive than DNS software vendors so I think in relative pain, IPv6 was probably a bit harder. But I think that because of mobile and other things IPv6 is now accelerating in use whereas DNSSEC is not growing quite as fast as I'd like.

AUDIENCE SPEAKER: It says here state your name and organisation and speak clearly into the microphone. My name is Martin Levy, I work for CloudFlare. What is unimportant is that my personal domain and my corporate company's domain is signed with DNSSEC. You are an eyeball network, majority. I speak as somebody who ships bits out on behalf of our customers towards the Internet. Between us, there is a common problem: The registrars, the registries. I would recommend, because it's unfair to do this, what I'm about to do, but I would recommend that they get brought up onto the stage and explain their role within this game and take the questions and the hits of that within this audience, not within, let's say, the safety of an ICANN meeting where they normally exist. As operators, we all have a common problem in the DNSSEC world, and that is the middle person. Now, our roles ‑‑ our position as a company has actually been stated on this, but I just leave this open for anybody else to either agree or disagree with me or yourself to make a comment.

PAUL EBERSMAN: Well, I'd certainly say, I have been tagged for doing a follow‑on Q&A during the DNSSEC Working Group and I would be happy to either give up our share that stage with the registrars to have this discussion.

FILIZ YILMAZ: Well, I can see you are here, I guess you can take this to him as well to them. Good day for you. Thanks a lot. All right, Eric, are you ready?

(Applause)

Eric is going to talk about the little Naughty project.

ERIK BAIS: Good afternoon everybody. I'm going to talk about our Naughty Port Project. And this started about, around December 2014, we were getting some hits from various DDoSs, we thought we need to do this differently ‑ customer‑driven, by the way.

So, a bit about what we do. I'll skip through this. We do IP registration, transit, provide Internet access in data centres. Some consultancy around that in that field specifically in these following Dutch data centres.

All right, so, this is the actual issue that we had, and it's interesting that these things tend to come in waves. So basically at sometime it's this customer and then it's that customer, but it looks to be school‑vacation‑driven, typically. And the issue is that it's not only on your infrastructure where you get DDoSs. It's actually your help desk. Customers start calling when their lines fill up, and basically they'll start complaining if they have packet loss, specifically if you have hosted desktop customers, VPN customers, customers that have gaming infrastructure. So, don't forget gamers, gamers, voice‑over IP operators, especially those, they really hate DDoS attacks. And they have little to no empathy for the fact that you have you have a DDoS on another customer because they simply say them, don't route them, kick them out of your network. And the issue is they are going to be up next the next day, so what do I do with them? And that's where the issue is.

Because, the issue in itself is not on the transit legs, because I pay my transit providers, I can actually call them up and ask them I got a DDoS attack and we see it in S‑flow, can you put an ACL for this customer for UDP traffic for here and there and make sure the lines are not filling up and we can use the rest of the capacity on the network. And the issue is on the Internet exchange platforms. So we were basically looking at how can we look at, filter out DDoS traffic, amplification traffic most of the time and see can we actually look into this? Can we look into where is this traffic coming from? Is it originating from a single source, is spoof traffic, we looked at Mac addresses, IP addresses, matched those up. We looked to see can we actually do ingress BGP 38, you don't don't want to know where we got started there. We actually looked to see if we could find hardware to actually do, you know, filter that into you know the equipment that we had. It's not that 50,000 ACLs, so there was no way that we could actually get into something that was workable on that process.

And it was actually interesting that the top speakers during the DDoS were actually not top speakers during normal hours and that was interesting. So if you look at this, this is how the typical internet exchange looks up. You have good peers and naughty peers and the naughty peers we define as peering partners on the internet exchange through the route server or direct peers that you typically don't do any traffic with unless you have a DDoS. And that's an interesting definition. Because, that will actually mean that we can actually take off those peers from that premium link where the green traffic is, and we divert that, then we may have solved the problem on the internet exchange. However, in order to do that you need to understand what we're doing and how they are actually participating in this whole process.

So what we did, you know and this is where the fun part starts, we actually started you know, a boot err site. You don't need to do that commercially, you can do that in‑house as well and you can basically download all the script from GitHub and you need to understand how does this work. So the traffic basically comes from one side and they send out traffic, you know, to the vulnerable servers, vulnerable providers, somewhere in a network and then at some point the spoofed IP address basically gets the target victim, IP address is here and this is where all the traffic is coming from. And if you look at this, you can actually see that certain traffic actually comes from certain countries, more than others. And that is what we found interesting. So let's go to GitHub, download some scripts. Be a regular script GITey. And this is actually interesting. It's actually very interesting to see how much of those scripts you can actually find on GitHub. It's actually very interesting. And then you can actually see how those tools are working, you see how what they use as input. There's no requirement to scan the whole Internet for vulnerable IPs, because you can actually download those as well. There is a search engine called Shodan and you will actually find vulnerable servers, so per device, per service that you want, you can get lists of IP addresses with all the specifications that you set per country, per AS, per open NTP server, open DNS server, whatever you like, it's a search engine specifically set up for this.

So, and that actually gives insight.

And this is what we actually found. During the DDoSs and you know after a while you basically say all right, we got through it. Now, let's see where does this traffic come from. And then we actually started looking at the sFLow tools on the AMS‑IX. AMS‑IX has a very nice customer portal. They actually provide sFLow graphs with traffic and per peer, you can actually see where it is that certain traffic is coming from, so you can see on your link, you do so much traffic to peer A, so much traffic to peer B, so much traffic to peer C. And during a DDoS you actually see certain patterns and we started listing everybody that matched during a certain time period, typically during a DDoS. So that's how we did it.

So this is the specific, we love AMS‑IX page because we could actually see this traffic here, we actually got the idea on you know, can we do more with this? So, once we actually did the traffic analysis on this side, we then started talking to the AMS‑IX nothing and asked can we actually divert traffic from those peers to another port through the route server, and how do we go about that? Can we actually have a naughty list and feed that in RPSL in the RIPE database and actually have the route server make the decision I want to include those peers here and I want to exclude those peers there.

So, thanks for Aris and Kostas to help us out on this. We then basically had to do the commercial stuff with the AMS‑IX stuff and say basically we want to test something out and they offered us the try and buy option at the EvoSwitch data centre so basically do this.

So this was basically what we did: list the ASs that were sending traffic and, basically, peer with those on the other side.

This is how it looks RIPE database. So, here, through the import via/export via, there is an except here on a specific list. So, we peer with everybody on the route server except with the naughty list. And you can pull up the naughty list as we have it today in the RIPE database because it's all public data.

So, that was the only thing that was basically what it took to get it done. The fun fact was, when we were ready to actually do this in production, we knew that AMS‑IX had two different vendors for the route server, and one of the route servers had some issues apparently to handle the roads of the multiple views because we were saying we want to have an include here, an ex include there, so instead of having a full view, it needed to create different views for different interfaces on, you know, our two different ports. And that actually was kind of an issue. Because, this was not just for us, and this was on a Friday afternoon and we said, you know, famous last words... what can go wrong? You know, we can do this, should not be an issue.

So currently AMS‑IX is a single route server vendor user, because one actually died and it didn't come back, it didn't recover. So, we asked the AMS‑IX guys, it's not in production yet, it's still testing, do we need to take it off line, off load this to the lab? And they said no, no, no, we want, it will be fixed anyway or not. So... in the end, you know, they went through the pain of the TOC cases and those kind of things, and in the end, you know, it was beyond helping them for this, so, trouble‑shooting helped, but the only decision was, you know, kill it.

So, kudos for the AMS‑IX, the easy option was to ask us to stop it, but they went through it.

Okay. So this is the result. So, we have the regular ports here, the regular peers, and basically say this is where we do the majority of our traffic and we have a choke‑point port here, and the choke point, the Naughty Port, is basically to peer with parties that have a higher rating, a naughty rating as we call it and I'll explain later how we get to that. And we basically do that through a different AS, and you can see that here, because this is a different AS number here than what our main AS number is here and that has to do with the route server. Because we actually have to tell the route server we peer with you here on this AS and we peer with this list on that AS. We couldn't do that on the interfacing on the IP addresses in RPSL, somehow it's probably a however implementation that's limiting us there, so we needed to do this with two ASs. But, you know, ASs are free, so, we just requested the second one.

So, this is where, this is the setup that we have. And here once the traffic actually fills up this pipe, it's not an issue, because the majority of our traffic is not here, it's actually here. So most of the customers are not complaining because they don't see packet loss.

I'll switch back to this later.

So, once we actually had this up and running for about nine months, we thought about, you know, giving this presentation and we thought you know maybe we can explain why certain customers or certain peers are on the naughty list. And can we actually predict who should be on the list? Because that was actually becoming interesting. Do we have Santa skills? Who is naughty and who is nice? And it appeared we can. So we can actually have... we can decide who should be on our peering list on the right and who is on our list on the left.

So, what we did is we actually created a database with every AS in that, you know, that had listings. We uploaded the aggregated number per AS, and we find those at Shadow Server, and we got aggregated numbers, open resolvers, NTP servers, everything per AS and we started pounding on the API for RIPE for how many IP addresses are they announcing from that AS number. And I have to give some understanding here, it's a high naughty rating does not mean it's a bad peer. It just means that they don't know yet, because we have spoken to networks in the Netherlands, for instance, and asked them about their rating, and they said, well, we run, you know, a good help‑desk, a good abuse desk, nothing is an issue in our network until we actually asked them to look at this and they said I didn't know I had so many open resolvers, I didn't know I had so many NTP servers participating in DDoSs, and because nobody told them, they never started looking for them, and the interesting thing is, you don't have to look for them. Because, if you ask nicely for the guys at Shadow Server, they will tell you on a day‑to‑day basis. The only thing is, if you want to do something with those reports, it's probably easier to automate your abuse messaging.

So, those ratings can actually be improved. And that's actually what some of the customers also asked us. They asked us: Yes, we don't mind if you, you know, include us or our ratings in this particular presentation, and by the way, how can we improve our score card? Because this is going to be interesting, we want to rate better than the other AS number on the same presentation.

So, if you look at some of the Dutch networks that I have listed here. This is actually a query that we did on the database. You have an AS number here. And AS name. The AMS‑IX organisation, in this case we're actually you know we're looking at this for AMS‑IX, so we just included the whole AMS‑IX list. And here is the naughty rating. So, basically it's a rating system that based on the announced IP addresses, and then how do you score? So, we have NTP servers here which is 63 on this particular network, and NTP has a specific amplification factor. DNS has a different amplification factor, and so there is ‑‑ in the back before the rating system, the rating of the amplification factor actually is included in the rating. So that actually certain vulnerable servers have higher impact on your rating, and also because larger networks, you know, you can actually have more vulnerable servers. So, it basically gives a very good weighted system.

So, have a look at here, triple IT, I'll get back to them later: 9.5. And typically our cut‑off rate for being on the naughty list is a 1. So, if you score higher than a 1, you become a discussion topic and basically we say you know, we should actually look at how traffic ratio is and is it actually worth having this customer or this peer on our premium peering port? Is it worth the hassle that they might inflict upon us during a DDoS? And that's how we're looking at peering currently. Which is different than just traffic ratios.

So this is the result for SURFnet, and this is the view from our parking portal where we actually made it very easier for customers of us that want to do this, and this is something that we will actually pre‑send in a specific website to the public as well. So, we will Open Source the data. We will provide this where you can just, you know, same as here, put an AS number in and get the results. So, that will actually make it easier for peering managers to review who they're actually engaging with. And on the interface, it will actually state you know in the pie‑chart, what the actual numbers are behind this. It's not just a pretty picture.

Rating is here, 0.16, announced IPs, and when the last update was.

Okay. So, how do improve? Like I said, abuse IO, it's free and Open Source software. Very nice to automate your message parsing and event handling. I can definitely recommend it. I have given a presentation on it on the abuse Working Group last RIPE meeting. They are doing a tonne of development. I can definitely recommend doing this. And then not to forget get your own report at Shadow Server, the URL is here in the presentation. If you need to look in, really look and search for it on the Shadow Server website, but if you just look, this URL, there is an e‑mail address where you can just send an e‑mail to, they will include this to you.

The presentation here says there is a backlog. They have actually put aside resources for all your requests, so within two days, all your requests will be processed. So, they have said, you know, tell them on stage we'll make this work for them.

So, on the Naughty Port, we can explain how this works. You know, if you want to do this, if you want to improve your rating or if you want to have, you know, a port like this on AMS‑IX yourself, we can at the moment you how we did it. We can provide the data so that you can actually make your own decisions, who do you want on your naughty list. This may not work for everybody. This may work for us, but this may not work for somebody else. We have a very Dutch‑orientated customer base and we do not have a lot of international geographically diverse traffic so we don't do a lot of traffic to Ukraine and Poland or Russia or China, so for us it's easier to, you know, compartmentalise certain networks and do not have an issue with that. That may differ for different parties. However, this may actually work for you as well if you are in a similar situation.

All you need is an extra AMS‑IX port, extra AS number for now and you can decide for yourself.

So, Triple‑IT actually did the work, and they are actually now down to a single DNS server from a customer and their rating is currently 0.00753. So, they are actually running currently from the top of the list, 9.5 in a couple of slides back, to the cleanest network in the Netherlands from this point of view. So, kudos to them.

Next up is we're actually planning to publish this information on the website. We're also looking into adding additional amplification factors to include in the rating system like, it FTP, because that's not in there yet. And we want to have AS sets as well so cannot only look at the announcing or the peering AS, but also at the AS set that they are announcing it from and see the top 30 from those ASs.

And that's it. Any questions?

BRIAN NISBET: Thank you very much Eric. That's most informative. So... please...

AUDIENCE SPEAKER: Ronan from the RIPE NCC. I have a question from a remote participant named Sascha Luck. He is asking: isn't the EU NET neutrality directive designed specific to prevent?

ERIK BAIS: No, because during normal operation the 1 gig traffic is ‑‑ the 1 gig port is enough for normal operation because typically we do like 10 or 20 meg to those peers so normal operation it's not an issue. It's actually during a DDoS when we get 70, 80 gig traffic, that's when it fills up. But then my regular port would fill up anyway as well. So, you are allowed also under EU regulation to act on abuse, and DDoS is in that regard is abuse.

AUDIENCE SPEAKER: Have you looked at doing this ‑‑ Avi Freedman with Kentik. Have you looked at doing this with transit? In the old days people would get a, it 1 to someone they didn't like and put their IRC server on that so that traffic would low slow that way. But some transit providers would allow you with communities say make me closer or further to these people, so you could pad all your peering and Oy ironically make yourself shorter to people in transit connections to people you couldn't do this with because you can't peer with them.

ERIK BAIS: Traits, there are multiple options to do that, you can look at FlowSpec, you can just no route traffic more than 500 Ecosystems away from you. There are several options. You can even nil route your traffic through your transit provider and still have your internet exchange filling up your complete backbone. So, this particular particular option is specifically for internet exchanges, but we're also obviously looking at what kind of garbage is coming in from the transit but that seems to be less of an issue.

AUDIENCE SPEAKER: I just have in addition to your list of the tools, just maybe useful to know that we have our router tool is absolutely free and anybody who authorised yourself, himself through RIPE database can check all vulnerable services in his autonomous system, all, it FP and check autonomous system before connecting to AMS‑IX or other IX, so probably it's useful to know that there is such possibility.

ERIK BAIS: Sorry, so, you want to have everybody?

AUDIENCE SPEAKER: It's possible for everybody to check their autonomous systems services before connecting to the IX.

ERIK BAIS: It's actual public data, so the information that we use is public data but only the aggregated data. People that want to have an impact on their own network and want to see what's in there they can request the data from the Shadow Server guys, they actually have the detailed data per IP address. We're not looking at that. We're just looking at the aggregated data per AS.

AUDIENCE SPEAKER: Exactly. You are looking on aggregated data but somebody who wants to use your service probably won't understand what's going on in his autonomous system and our tool provides this possibility.

ERIK BAIS: So that data can be requested from Shadow Server for free.

BRIAN NISBET: If it's a very quick question, then question.

AUDIENCE SPEAKER: Mikkel Kristensen from One.com. My question is, do you have any problem with the BGP to the route server going down when you have attacks?

ERIK BAIS: Yeah, because during the the DDoS attacks we don't have an issue keeping the sessions up to the route server, that works fine.

BRIAN NISBET: Okay. Thank you very much, Erik.

(Applause)

So, now we have three and a half lightning talks, which will all be very quick. And up first we have fill Roberts, who will be speaking about, it's a Cryptech update.

PHIL ROBERTS: I hope my lightning talk isn't the half one...

I am the newest person helping out on Cryptech. My role in Cryptech is to do what I call business development. That is not fundraising, I'm not a fundraiser, but I'm working with people to find out what requirements might be for an eventual product made on Cryptech and I'm helping to find people to help build such a product. So I'm going to give you an update about Cryptech.

Cryptech. What is it? It's an Open Source, hardware security module set of reference designs. There's both hardware and software reference designs. Hardware Security Modules are used to manage keys basically. These things exist to as stand‑alone devices or as cards riding on servers, there is a bunch of configurations for Hardware Security Modules and they are used in all sorts of operations.
So, Cryptech. Why should the community care? Why should anybody care that we're building an Open Source version of this? There is a lot of concern in the community about compromised devices, routers, servers, firewalls, various kinds of things that people worry that some large Government entities have put back doors inside. And when you talk to people who use HSMs and you say are you concerned about this? They say we hadn't thought about it, but yes, it is a concern. We don't know what is in our HSN, we are buying this from somebody and we don't have a lot of assurance that they haven't done something that we are not comfortable with in their HSN. When the Internet community thinks about this and thinks about securing key Internet infrastructure whether it's a DNSSEC or RPKI. We would like to be in a position where we have a better assurance where that tool hasn't been compromised. So some of the leadership got together in the fall of 2013 and said wouldn't it be interesting to build an Open Source version of this. So that the protocols that we build would have a better assurance of operation on hardware that we can trust.

So, what's the current status of Cryptech? We have working Alpha Boards, a couple of weeks ago the first Alphas were delivered to the engineering team. They have gone through bring‑up, they have gone through functional testing on all the key components of it. And the Cryptech Alphas are live and it went well enough to order a set of Cryptech Alpha Boards to share with other people in the community who interested in being Alpha testers. This is a picture of it. This is the bottom. This is the top.

But there it is, it's working. I think that the community of people who have worked to get it to this stage ought to be very satisfied by their accomplishment. I think it's quite a good achievement.

So, why am I here? I'm here seeking Alpha Board testers, so if you use, in particular if you are using HSMs in DNSSEC or would like to use an HSM in DNSSEC, I would like to talk to you. Cryptech is configureable. There are a set of different configurations it can be built to do. The first one that we are expecting to use is DNSSEC. So, if you are involved in DNSSEC operations and would be interested in being an Alpha tester, I would love to talk to you. Shortly after that, we think we'll be able to support RPKI operations so if you use HSMs in that, or would like to, I would like to talk to you.

And we're interested for people who have implementation and operational experience who will work with us to do functional testing on this.

How else can you help? There are a number of other ways that we're interested in getting community help on. Community review, there's the URL for the pointer to the resources, so if you are interested in being a reviewer, we would love that help. We need help on testing like I said. There is an interest in having to help in documentation, on what this thing does and in particular if you're interested, if you work for a company who might be interested in building a product on this, I would love to talk to you.

And finally, we're interested in funding support, like I said I'm not a fundraiser but if you are interested in funding this, I would put you in touch with somebody who would be happy to talk to you. The last slide, who are the supporters? I'm not sure this is a complete list, but I pulled it off our website. It was the logo of people who have provided financial support to this project so far.

And that's all. Any questions?

BRIAN NISBET: Are there any questions? We have a moment or two. No... you are all fully convinced ‑‑ wait... someone suddenly decided they had a question.

PHIL ROBERTS: I'll be here through Wednesday.

AUDIENCE SPEAKER: Sebastian Castro. Can you clarify what kind of interfacer you have for the ‑‑ I saw it's USB?

PHIL ROBERTS: It's a pair of USB ports running PKCS 11. I think the panic button is there for testing tamperproof stuff. I think.

BRIAN NISBET: Okay. Thank you very much.

(Applause)

So ‑‑

AUDIENCE SPEAKER: I just have a quick question.

BRIAN NISBET: Can you talk afterwards. Next up, we have Lu Heng. Lu will be speaking about invisible hijacking.

LU HENG: Hello everybody. Just a short story about an issue we currently, we recently faced at the beginning of the year.

It's called invisible hijacking. We started in 2008. We went /manage infrastructures, and we also solving software solutions.

What happened? We started receiving spam cop report for our range about half a mill in size and it was about two or three reports a day, from the same reporter, so we look into it.

And what we did: we ‑‑ because we don't know if any of our mission was compromised so we basically removed all the mail servers which have been in the range and block all the mail ports in the range. So we moved basically the whole range into a dedicated VLAN and made sure there was no service around that, from the VLAN sending out any single e‑mail. But we still, we received spam cop reporting, and it's kind of weird.

Then, we announced the range into a different data centre, with a totally new set of server. We still received the spam cop reporting. Totally impartial because, I mean, if one of the servers was hijacked, or was compromised, then we shouldn't be receiving more spam cop reports, because it's into a totally new data centre, so it's impossible for them to compromise two data centres at the same time with same hackers.

So, we thought our IP was hijacked. Then we checked the global routing history and no unknown announcement.

And we talked to our network provider, which is MTN in Africa, unless the whole team in Africa is a bad guy, it seemed very unlikely as well.

So we called Yahoo, because all the spam cop report was received from Yahoo mail, so, we called Yahoo, Yahoo showed us a range which is /18. We have two announcements, one is a /12 which is our total allocation and one /16, which is a new announcement which is we move to the new data centre. So, /18 doesn't exist and as a /18 was never existing in the global routing table as well.

So, apparently Yahoo received this /18 announcements from direct peering in DECIX of a dead AS. That AS was not seen since 2010. And that's how it looks like.

So, upstream the bad guy? Maybe yes or no, because maybe someone leaks the information about our range and saying, there is a large range being announced here, maybe you can use it, but we're not really sure about it. But technically, we guess no, the AS number was dead since 2010, and our network provider is not peering with them for sure.

So, who is this AS 17445? It's belonged to a nonexistent Chinese company. It's always announced hijacked range, all this history basically, and its parent company, when they are using this hijacked AS, at the same time, they also announced an AS number in DE‑CIX from AfriNIC free pool. The AS number was not even assigned yet.

So, the DE‑CIX customer apparently, DE‑CIX bought through reseller, and at first check, the AS number seemed legitimate, but they changed the AS number right after they became the DE‑CIX customer. So, that's how it started to happen.

So in summary:
What they did, the spammer, they become exchange customer. They do a direct peering with a free mail provider like Yahoo or Google and announce a hijacked range in a more specific announcement through direct peering. And by that, it's basically all they need to do is dump the e‑mails through direct peering into all this free mail providers without being even seeing the global routing table and because 90% of mailbox in the Internet is essentially free mailbox so you can reach almost everybody on the planet without even being seen by anyone.

Prevention? This problem is very difficult. Because, it's difficult to find out in the global routing table, and if you are a large organisation which has a large range, it's difficult for you to pay attention to two or three spam reports a day, this is essentially nothing for most of the large ranger holders. And it's difficult for exchange to identify a customer like that because the exchange account really regulate private peering. And it's difficult in the peering part as well if you are not doing RPKI.

How we find out? Because the speciality of our service we treat spam cop report very seriously. So when we receive them we really look into it, we pay attention to very unlikely report. And how we fixed it: basically, the social connection we have to Yahoo and DE‑CIX guys, we called up the network team in private and say hey, guys, what's happening? And otherwise, I mean it's almost impossible just to reach them by general lines.

The future:
DE‑CIX promise to check customer's AS after they changed it in the future. And RPKI seems to be the only visible solution to a case like this because then it can totally prevent somebody hijacking through private peering, but of course there's more debate about the solution other than the specific case here.

Any questions?

AUDIENCE SPEAKER: Hello, this is Wolfgang from DE‑CIX. Actually, we have always checked the AS numbers of customers also per AS number changes but the only thing we can check is what is in the RIR databases and if this entry has been falsified, we have no way to validate that.

LU HENG: Actually, it's quite obvious in this case because one of the AS number which was announced in DE‑CIX was in the free pool of AfriNIC, so, it's not existed.

AUDIENCE SPEAKER: It existed in AfriNIC, I personally checked. And I have actually rejected that the change from AfriNIC and they came up with another AS number from APNIC which also existed in the RIR database of APNIC. So, it existed every time. I cannot see if the entry in the RIR database is actually falsified. That's up to the RIRs to put on some sanity check before an AS number goes into the RIR database.

LU HENG: I do talk to a registration services manager in APNIC, and the registration manager managing the APNIC case said it was hijacked by someone, so, it's ‑‑ but in AfriNIC case, it was an unassigned undelegated pool, so it was pretty clear that the AS number was not used.

AUDIENCE SPEAKER: It was in the database but I actually rejected the change. But then they came back with the APNIC AS number and I couldn't reject that because formally it was okay.

FILIZ YILMAZ: Sorry. Just a piece of information. Thank you, Lu. Also for the question, but I have been just informed that this conversation, this topic will be also revisited at the Anti‑Abuse Working Group on Thursday, the second slot. So you may want to pay attention to that Working Group session too.

AUDIENCE SPEAKER: Ruediger Volk. Deutsche Telekom. This shows that best practices, let's say, authentication of BGP sessions should be done by MD5 and that should be sufficient, obviously are not getting the actual problems that are there. When Lu says, well, okay, RPKI could help, I would like to point out that at the moment, the RIPE NCC RPKI cannot give me AS certificates and actually I would need the delegation of those for doing things with it.

LU HENG: But in this case, the certificates for the prefix itself is sufficient to prevent what happened. But you're right.

RUEDIGER VOLK: Well, okay, for authenticating that the peer that you are opening a BGP session with is not sufficient. And in the end we actually all should take care of also authenticating that your peer is actually the guy who is telling us he is.

LU HENG: Yes, I understand that.

FILIZ YILMAZ: Okay. Before I give the mike to the next person, I have to say, I will close the mikes now because we are out of time and there are already three more speakers ‑‑ sorry, commenters on the mike. Yes...

AUDIENCE SPEAKER: Hi, Ruediger, this is specifically for you. It's Alex from the RIPE NCC. Getting AS numbers on your certificate will happen at the next signing session. All the code is done. We just need to go to the data centre, sign it and then you have it. All right?

AUDIENCE SPEAKER: Hi. I think the problem, the exact problem is that anyone can, at any AS number into its AS macro in the RIPE database, so maybe it would make sense to anyhow authenticate AS numbers added to the AS macros in RIPE database, because ‑‑ it's not only about route servers, it's also of course about direct peers too. Thank you.

AUDIENCE SPEAKER: Hi. Elvis Velea, V4Escrow. We have actually seen something similar happening for a couple of our customers that have done transfers in the past few years. I was not aware that this is happening also for address space that is has been registered and announced. This was happening for address space that have changing hands and in the period where the address space was not announced, although it was not showing as being announced globally by anyone, spam reports were being received, again just as you said, mostly for Yahoo. We did also realise that this is happening because the address space was being announced and somehow was accepted by Yahoo through a private peering. So it would not be seen anywhere in the global routing table. Now, what happened was that this, in our case, the address space team had route objects saying please allow only this size number to route this prefix, but they were ignored by the ones that were applying filters and somehow they were just reaching Yahoo and...

LU HENG: If you look at this slide, it says, 16637, it's encounter network, so when they announced through the direct peering to Yahoo, they put our network as their next stop. But obviously they are putting a fake machine, pretending to be our network, but from a Yahoo point of view that it was from authentic network.

FILIZ YILMAZ: Okay. So, I think this is ‑‑ thank you, Lu. It created a lot of attention, as we expected. Thanks a lot. And I think you will have more to talk in the corridors as well as the anti‑abuse Working Group. So next speaker up is Geoff Huston. And normally, we are out of time. But we still have one‑and‑a‑half lightning talks to go and Geoff's is first. That one, about zombies.

GEOFF HUSTON: Afternoon all, I'm Geoff Huston, I'm with APNIC. Actually, this is a relatively special crowd on the Internet. Because a fair few of you are responsible for the way that DNS works today. And, by God, you are doing a shit job, aren't you?

Here is what we do: we try and abuse this. What we do is run an online advertisement and this advertisement does a really simple thing: it just collects one by one blots. Now, when we're trying to measure the Internet, I don't want to see your caches or your middle ware crap, or all this rubbish that you pile into the network. I want to see what users do. So to do that, I use unique DNS names. So, we run about 8 million of these a day, of these ads, and each ad generates around three unique names per day. We get at least 24 million DNS queries for absolutely unique names. Unique. That name never existed before now. And should be used once, and only once. DNS folk, listen up: once. Okay. So, similarly, too, for the web‑caching lunatics out there: once, that's all. I never want to see this again, ever. This is what 'unique' means, or at least that's what it means to me; it may not mean that to you, some other special meaning.

The idea is, I send one machine a unique name. The TTL is one second. It should fetch that unique web name once, because that's what the meaning of the word 'unique' is. And so back at my authoritative name server, I should see one query. One. And that query should happen pretty quickly, shouldn't it because the DNS is fast because you guys are professionals, yes? So here is what I see, oh professionals. Now, if any of you have are really really good at unique time stamps, you can guess what's going on. For the rest of you, I'll translate. That column at the left is when I saw it, in this case I saw it on the 15th December last year at 3:54. But what's that column on the right? That's the time I generated this unique name, the time you should have been asking this name. Now, for those who are really quick here, you would have noticed it's about a month old. So, somehow, some of you are sitting on this DNS name for a month, waiting it to age properly or something; DNS names improve the longer you sit on them before actually asking me for the answer. You guys are nuts.

So, is this rare or common? You know, how many theories do I get that are zombies and how old are these zombies? So I started looking, and if you think DNS gets better with age, by God you are doing well because some of these queries are up to three years old. That's a cumulative distribution. Half of them are more than six months old. Well and good. But I tell you, the DNS goes shit after three months. After three years, the answer is completely irrelevant, don't bother asking me. I don't care any more. One DNS server massive zombie load. But I run three servers. And all of them got precisely the same profile almost half of these zombie queries, the ones that are old, are more than six months old. I know DNS designers are obsessive compulsive. And you know, I don't mean that personally, Mr. Andrews, or anyone else, I don't mean that personally, but you are. Totally obsessive compulsive. And to say that you should forget about queries is a helpful comment, forget about them. Don't hang on. So, I don't know what you're doing. But when I look at this over 180 days, over all my DNS servers, 72 billion queries, 22 billion are zombies. I don't know about your authoritative name server, but mine, one third of the load is shit. One third of the load is complete crap answering queries from the dim dark myths of pre‑history. Now this is getting really, really weird. You guys are insane. More to the point, you don't even do it once a day. Some of you are doing it 12 times a day, 24. Some of you are obsessive and doing it up to a million queries a day for the same name. Get this: It never changes. It was always the same answer and asking me a million times won't change that. It's the same bloody answer.

So, now I'm thinking, are you DNS guys really that inventive or are you just tagging along for the web? Is it the web guys are the bad people here and the DNS is just being roped in as the victim. So let's look at web zombies, that's the distribution of age and although there are zombies out there in web land ‑‑ bastards ‑‑ you have got a completely different distribution. 50% of the zombies are less than four days old. The repeat rate is entirely different. These are different zombie factories. Totally different zombie factories. So you know, I can get back here and go, no, you DNS guys aren't innocent victims of some evil web thing. You are your own zombie manufacturers, you are deranged. Completely and totally.

So, you know, no bad deed goes unpunished. If you are on this list, you are part of the problem. And kudos to a DNS resolver in Guatemala, 4.6 billion zombie queries were generated over this period. If they have shit Internet in Guatemala, let me tell you it's not the bandwidth. No way. There are also some other interesting folk there of all kinds, including team Kimro, so, I'm starting to get a little interested about this. What can I say about it? Well I actually think there were three zombie factories, because when I look at the zombie repeats, one class of folk are stalkers. We know all about stalkers. What they do is they look at what you do and then ask again. The stalkers ask a lot of questions, but each time they ask only once or twice. Then there are the storers, the folk who say, ah, you did a TTL, a time to live of one second, that means I have to reask every second, don't I? No. The time to live was a death time not a re‑ query time. Get it together. And right on the far right are the totally deranged, who manage to repeat queries up to a million a day because they can. So, let's have a look at the stalkers now. That's the list of the stalkers and team Kimro is right up there. Blue coat systems, Liberty Global in the Netherlands, if you are on there, we can see you, we know what you are doing, thank you very much.

Here is the list of the deranged. What a list. There is our friends from Guatemala, more net in the United States because they are an educational network and they know what they are doing. And if you see yourself there, and if you see your IP address there, for God's sake do something about it because all it's really doing right now is just asking zombie questions and they are not appreciated.

We can use this. No bad deed should go unnoticed. Because it seems that the DNS never forgets. So if I want to store something, just make it into a query. Turn the data into part of a query, because reading is really piss easy, you just listen and sooner or later your data is going to come straight back to you every single time. About the only thing you can't do is delete, because the DNS never forgets.

Thank you.

(Applause)

FILIZ YILMAZ: Thank you, Geoff. There seems to be one question.

AUDIENCE SPEAKER: You thought you were going to get off easy. This is Olafur from CloudFlare. You are yelling at the wrong crowd. You should be yelling at the big data people. They are the ones that are delivering most of these, they are people replaying old traces and logs. I get a lot of these questions, because people are just repeating queries from their traces.

GEOFF HUSTON: That's the stalkers and that's the folk that I can see that are just replaying the logs. The deranged folk, there is something really weird going on in their DNS and I really don't know what they're doing, but yes, stalkers and relog replayers sit there as well. The DNS is amazing. No matter what you look for, somebody is doing it. Always. Thank you.

AUDIENCE SPEAKER: John Curran, ARIN. Before I invest in your DNS storage business, I am curious, have you actually looked to see what percentage of your queries come back? Is it 5% of your queries are in that space or do you actually get all queries are zombies, it's just a question of how much?

GEOFF HUSTON: John, that's a really good question and I'll be back in three months to give you the answer. Thank you.

(Applause)

FILIZ YILMAZ: And here comes our half‑lightning‑talk from Patrik.

PATRIK FALTSTROM: Thank you very much. From Netnod, also a Chair of the Security Stability Advice Committee of ICANN. I want to make all of you aware of this vulnerability that was announced at 4 p.m. local time here today.

There are a couple of things that we, of course, know about main collision issues, about new TLDs that are released and that people have been using the TLDs internally in their companies without them actually existing in the route zone. We also know about the WPAD protocol that is used for auto configuration of proxies for web browsers specifically.

What is happening is that a lot of people, including specifically DNSSEC is that if it is the case that you are using a TLD that is not in the route zone an then suddenly it ends up in the root zone, in that case weird things can happen. Specifically what happens is that you were earlier, when you used this zone, relying on an NX domain response coming back when the query hits the root sever, if it is a case that the SLD existed then you get back the records, you follow the chain of delegation and in the case of WPAD, as I will explain on the next slide, it's actually the case that someone might actually have put up some data on the zone ‑‑ in the DNS for the domain name that you thought did not exist globally but now suddenly exists.

Another thing that I want to everyone to know that the WPAD protocol is not only used in the Microsoft products, it's now implemented in many operating systems and brothers, it's not turned on by default but many people have probably turned it on. Turn that off.

So, what we do know is that we have this risk for collisions, we talked about certificates, internal certificates that have been talked about here a couple of years ago.

But, what we don't know is how much this is news, so there have been some work going on trying to look at the logs of the actually queries that are happening. And there are two things that you should be aware of. First of all, for those of you not really knowing about the WPAD protocol, you see that at the second ‑‑ if it is the case that you internally use something that, historically, was not a TLD ‑ for example, Corp ‑ what is then happening is that your brothers will try to fetch the file wpad.dat from wpad.something.corp, and what exactly the second level domain is something that depends a bit on what kind of auto configuration we're talking about.

The scary part here are two things. First of all, we see more than 20 million lookups in the wild of these domain names that has WPAD in it. And the second bad thing is that people have started to actively register domain names in the new TLDs and put up WPAD files on their web servers.

So, what these people are doing is they actually put up these web servers and they are sitting there and just waiting for someone to issue DNS query, the attacker is then serving the WPAD file that tells what proxy to use and then all http traffic is going through that proxy that people have set up. The only way to get around this is to stop use, as many of us have said, stop using anything else than fully qualified domain names. Only use fully qualified domain names and TLDs that actually exist in the root zone and stop using WPAD traffic and you better look in your firewalls to see how much that is in use in your organisation.

There is a paper, there is quite a long URL, so, just sort of shorten it down so you can look at the paper. The attack surface is much much larger than any of us expected, just a few months ago.

Thank you.

BRIAN NISBET: Thank you very much. We have no time for questions. So, that is the end of the session today. Please remember to rate the talks, nominate for the PC, all of those other things. Jan is staring at me so I presume there's something later I should mention, BCOP. There are other things on this evening and then there is the welcome drinks later on and the Plenary restarts again at nine o'clock local time tomorrow morning. Thank you all very much.