Plenary session
24 May, 2016
At 11 a.m.:
JAN ZORZ: Can people get their seats, we will start in a few minutes, if we will a bell to bring people in, start ringing the bell. OK, let's start with the sort of like IPv6 session. My name is Jan and together with Shane, we will do the housekeeping of this session. Remember, that it is recorded, and webcast. Please also rate the talks at the end because it's quite important for us, from RIPE Programme Committee, to understand which sessions and which talks you really liked. There is a PC re‑election coming up, so please also volunteer for the PC. And I would like to ask ‑‑ I would like to introduce my dear friend, John Jason Brzozowski, coming over the big pond from Comcast. Honestly speaking, I was chasing him for years to come over here and tell us what they did on IPv6. This man single‑handed deployed IPv6 at this big network and was chasing people around and yelling and screaming at them that they must deploy IPv6 fairly early. Thank you for coming, John.
JOHN JASON BRZOZOWSKI: Thanks, Jan, I appreciate it. It's my pleasure to be here, this is my very first RIPE, so you are all going to be on Facebook later. So, let's get started here. So in the coaching that I got coming in preparation to come here, it was difficult to decide exactly how far back to go, so we will go back briefly, ten or eleven years, and then if you all have questions about kind of historically where we have been and such, then don't be shy. From what I have gathered so far I don't detect a great deal of shyness in the room so I don't imagine that will be a problem.
2005 is when we started. One of the corner stones of the programme that was basically pushed out from the top was this has to be seamless and what that means is by deploying v6 we can't break anything, in particular the customers' experience and kind of the way that we run the business, and we have lived and we live by that for the, for the programme thus far and it remains cornerstone of what and how we do things for v6 at Comcast. V4 was not enough room four years ago, we were out of space, we couldn't fit it and were our view v6 was inevitable. So Jan and I were talking, going back and forth with e‑mail and the question came out: Why? I think that is pretty straightforward, if you need anything more around why we can have that conversation but I think it's really that simple. And the scope, the scope is basically everything, right? The routers, data centres, cable modems, access, video, we are going to walk through in fastforward fashion what an incremental version that have looks like over a ten‑year period of time.
So, the first v6 only service, if Paul was here from Facebook he would throw something at me because he says hey man, stop using the same slide but it's not the same slide, this gets updated every Friday and I get a copy of it. What you will notice here is 98% of all the K nodes, used to deliver services to our customers are v6 only modes, no v4, 98%. You will notice that this graph, the left‑hand access is millions, so we are on the can you say up of 40 million devices. And I think that is a lot by any standard. And this modem management network alone, which is really mostly for almost exclusive for Comcast use, most of ‑‑ none of our customers see this and the rest of the world doesn't see this, it alone is a massive v6 deployment all by itself. And of course, what we are trying to do right now with this part of our deployment is, we are trying to gaining this towards 100 percent. If you look here over the period of time here, looking back about three years ago, this ‑‑ these two lines crossed dramatically, and you will notice that where that green bar is, that would not have happened without IPv6. Those are new modems that were added because of v6, v4 could never have supported that in our network. So, again, going back to some of the discussions that we had kind of preparing for this talk, why? What was the motivation? That was the motivation. I think that's fairly straightforward from a visual point of view.
What else? So, we decided we enabled v6 on the broadband network, on the backbone, the regional networks, etc.. we did modems first, and frankly, on the front end of our deployment the, all of you all on the Internet really didn't get to see a lot of the fruit of that labour because the modem management network was really internal for Comcast folks to use to support our products and services. But what we did is we decided to use the work that we have done, not reinvent the wheel and used it as a building block or stepping stone to enable more and more things for v6. So next step for us logically speaking was to enable broadband. So, care of Geoff and APNIC, thank you, Geoff, we are addicted to your graphs, we look at them every day but this is basically a picture of our deployment from APNIC's measurement point of view for however many of years Geoff goes back. So, according to Geoff, that looks like it's about 60% and some change that he season a fairly consistent basis. But from our point of view, what we see is about 87% of our customers today are enabled with native dual stack broadband. So 87% of our customers something to the tune of 18‑and‑a‑half, 18 .7 million customers are enabled with native v6 today.
We think, hope, I don't think it's really a hope but we estimate, that by, say, the first quarter of next year we will get that number as high as the low nineties and at that point it's people who brought their own cable modem and their own retail wi‑fi router, that we will have to work with, to make sure we get those devices to support v6 by default. 87%.
So over the years, one of the other things we have been talking about is X1, which is our set‑up box and next generation video platform. And it's been a long time coming, for many of you who I know in the audience you have probably heard this on multiple occasions over the past more than few years. One of the things that we are very happy to share with you is, really this year, this got kicked into high gear and you will notice this is about a six‑week period of time where we migrated 40% of our entire box deployment to IPv6 only, so you will notice that this one is v6 only. By June, that will all be done, something to the tune of 15 million, will be done by the end of June, say, early to midsummer. That is a big move for us. It is ‑‑ if you Google anything with Comcast and video our Comcast X1 it is our flagship product so going back to motivations and belief in v6, if there is any question by anybody here as to does v6 work, what do you use it for? Why would you deploy v6? We have given you three extraordinary examples of why and how we are using IPv6.
So now what? So some of the other things, not to make this kind of a cheerleading kind of ra‑ra marketing pitch for v6 because maybe that is boring, hopefully we have shown you it does work at scale and frankly this hasn't been a gift; this stuff mostly did not work out of the box, we had to make it work so one of the things I was asked to do was share the good, which you have seen, kind of the bad/you goly, we can dive into that conversation towards the end with Q&A but the thing had a we noticed early on is, yeah, there were some core router platforms that did the things we immediate for hem to do but by no stretch of the imagination that we have technology readiness across the board. I cannot count how many conversations we have had my router does not do this, my router software version, I have to upgrade this hardware this, we didn't have the choice, so we had to bite the bullet and make it happen. You name it, everything from router switches, firewalls, seem TSes, for people who are capable, they all needed an enormous amount of it. LC and in some cases, very unpleasant conversations, incrementalism: One of the distinct benefits that we have had over the past ten years is, we have been able to push say 1,000 ‑‑ a millimetre a day, one could take we have taken incrementalism to the Nth degree but at the same time if you think back to the very first slide and the fact that we wanted v6 to be seamless and we needed it to be, incrementalism is the only way to go. One of the things that we have said to people over again, the longer you wait, the less opportunity you have to deploy anything from ‑‑ incrementally, including v6, so for us incrementalism is a word to live by.
And one of the things that I could tell you that I have become pretty good at, having a plan A, a really good plan A, that is a good idea, but you should have a really good plan B, C, D, perhaps all the way down to Z. Can I not begin to tell you how many change saws we have had to juggle and the acrobatics we have had to do to make some of these things go and I think if you don't plan for the deployment of a technology like v6 that is so pervasive that way, I think you will be surprised. In an unfortunate way.
Bugs. Bugs happen. I don't think I am telling you guys anything you don't know. Some bugs are minor, they are cosmetic, others are severely, they are severe and operationally impacting. Some of the earliest v6 deployment we had, and I think you guys know Lorenzo from Google, he was one of my first customers and he is th eternal squeaky wheel, and he bore the brunt some of some of the early bugs we had, painful more so for me than him but they do happen.
And everything is IPv6's fault initially, even today. We have something go wrong in the network, it's v6's fault. We are finally getting to the point now where very positive things are the fault of v6, including things like these graphs you saw where we are protecting the business and allowing various aspects of it to grow, almost in an unfettered manner. And it does work. For those of you who may or may not believe, it works and it works very well and it works at scale.
So, just to kind of use this as a Segway, like we said, we ‑‑ one of the things we didn't do was waste our time deploying point solutions, redeploying things and reinventing things we tried to keep it as much as possible focused on building blocks, we enabled it on the network, we used that to deploy something else, back office components data, centres, access network, you name it. Everything has been a building block for us. And we have worked it from the centre out, towards the edges, and we really did work on the part first that none of you care about, but it cared ‑‑ it cared the most ‑‑ we cared the most about it and the stuff that mattered to the business the most, first. We remember back in the days of NANOG 46 when people said, when is Comcast, when are the cable companies going to deploy v6, we said it's coming. Those were the days where we were ‑‑ we are chomping at the bit to do this work but we had to put the business first and the cable deployment management ahead of everything else. If we didn't do that, unfortunately the side effect would have been many other things would have suffered long‑term.
And we have talked to a lot of people around the globe, in this community, North America, you name it, a lot of people ended up deploying site solutions to enable v6, a separate provisioning platform, this and that, we basically stayed the course and we deployed stuff singularly so the provisioning system that was serving existing customers, we upgraded it, over two‑year period of time, that same infrastructure was enabled to support IPv6, we didn't' have a separate dedicated side infrastructure. And frankly, one of the things we have also done is deployed v6 per service, one service at a time, right? We want to make sure that we had it right and it was working well before we moved on to the next thing. One of the other things we are very proud of is, we deployed v6 to our customers one time, we did native v6 the first time, there was a lot of options on the five years ago, and I think many adopted those things, they made the right choice picking those technologies up but if we look back ten years ago and said we helped to develop v6 for DOCSIS and cable networks, we knew we were going to invest the time, and energy and the blood, sweat and tears to make the solutions ready for for v6, we are going to deploy native v6, and I would be lying to you if I said it was a walk in the park because it was not, there were early days where we had bugs but they were looking back now, five, ten years ago, it was worth every minute.
So what is next? Where do we go from here? I mean, I am wondering how many people think good job, go home now, go to happy hour. You are done, right? No, we are not do. Right. So we have talked about turning on IPv6. We didn't really turning off IPv4. And that is really what happens next. So, hey. So we want to continue down the path of minimising our dependencies on IPv4. When Shane was up doing his talk I was happy to hear many of the mentions that Shane had for v6 and how important it is for I O T. We feel the same way. From our point of view there is nothing that v4 stands to offer us long‑term, as a matter of fact it's complicating matters for us. So our goal is to continue to go down this path and minimise and reduce our need for v4 long‑term, to the point where we just turn it off. Think back two, three slides, 98% of 38 million modems are v6 only, it's not that long from now when they will all be v6 never to see a v4 address again.
I talked about that one just now.
Our Internet traffic today, 25% and growing v6. We expect to see that double this year, right. By the end of this year, we estimate that traffic is going to be 50%. If you look at the moves made by Apple, Facebook, all the folks who have been pushing V6 for a long time, especially the Apple announcement for the iOS applications in the AP store, those are moving activities, they are significant, those are the things they are going to help to us where we need to go. This is very vibrant and inter‑related ecosystem, so we all work very closely with one another. And of course, our intention, not only just to turn off IPv4 but v6 for suss a platform for innovation. It represents the future platform that we can use. Imagine, I don't want to plagiarize Shane that much, if you think back over the years, the new things popped up, programming languages, techniques, this that, there is always something new that we used to build the next generation of our platform on, for us v6 is that platform, we want to make the network smarter, more efficient and basically a tool for us to build future products and services.
So the path to v6 only. What does that mean? So, and why? Why can we say that today. So this chart here is basically a graph of link utilisation for v6 with a large search company that everybody probably uses today on the Internet. You will notice that almost all the links have more than 50% utilisation for IPv6. And this is going in the up wards and to the right direction. This is Facebook's. 60% of what we see from them or what they send to us is v6 only and growing. So, did anybody imagine that it was that high? You know. Well.... really for the audience, there is a lot of people who see that, they are like, really? Yeah, really. And like I said, when we look at our over all network traffic we see the trend going in that same direction so why wouldn't he have the conversation now what life after v4 looks like, what turning off looks like. What does it look like for us? We manage the network entirely using v6. So I'm sure everybody in this room logs in their routers, configures them, does things with them. In our environment and not just in future that is all going to be happening over IPv6. Aside from automation, the transport will be v6. We are looking at innovative approaches to zero touch provisioning of routers and the network for those people who run very large networks, you know, like I think of conversations with the Googles, the Yahoos, the face books of the world where there is so much gear being bought and deployed on a daily basis, we have the same problem, so think about technological approaches that can be used to automate the deployment of that infrastructure all using IPv6. And of course, for us, if you look ‑‑ think back to the slide that we just passed, the majority of the traffic on any one of those links is v6, is majoritavily v6, why wouldn't we start talk about v6 only network Czechs. And if you think back to the X1 deployment slide with the up and to the right dramatic up and to the right, and the fact that our X1 platform is a huge customer of our Cloud internally, we are looking to make the Cloud v6 only. I don't know if Tore is here? So Tore Anderson has done a lot of work in S.I. T and some other stuff for making the Cloud v6 only capable, a lot of interesting technologies in that space are things we are looking at to further alleviate the need for v4 in our network. And of course, last but not least, broadband, broadband is most certainly going to be one of the last things that will go v6 only but we are already looking at technologies and approaches, friends and family trials and things that we believe that represents the very interesting intersection of work that we could do with this community, so we have v6 to 87% of our customers today. Why wouldn't we use that transport, it's good enough for X1 and to run modems that provide broadband and other things, why wouldn't we carry v4 as a service over it, so for those of you who are familiar with the v4 as a service technologies, MAP, GRE, you are talking about the idea where everybody in this room knows off v4 address on your wi‑fi router today, v4 address means you no longer have that and your LAN v4 traffic, which by the way is declining, will that be carried over v6 to some other device like MAP B R or what have have you to get to the Internet. A lot of these concepts, common theme here, will be useilised over and over again for things like wi‑fi, for those of you next generation access technologies like PON, DOCSIS, 31, and of course not to be kind of telco and irritate Shane here, but we have a very large phone deployment as well, all these devices will go v6 only for us. When we started this talk off and we said the scope was everything, the scope is really everything, right? To the point where we are now starting to talk about turning off IPv4. So I have some backup slides, Jan gave me a card that says there is ten minutes left. I can go go into these unless people have questions or comments. I would prefer you guys talk to me and have some questions or comments.
JAN ZORZ: So I need to keep the queue.
AUDIENCE SPEAKER: Leslie Carr with S F MIX. So, you may know that some major virtualisation platforms and CDNs don't currently support v6, do you think there is anything that Comcast or any other large market shy can do to encourage them to support v6? JOHN JASON BRZOZOWSKI: Yes.
AUDIENCE SPEAKER: What is it? JOHN JASON BRZOZOWSKI: We are customers of some of them and to be perfectly honest with you, I mean, we are not delusional, we recognise that we come as a customer, we have certain needs, if you look back, Leslie, to this slide here, this one here, the X1 slide, a portion of our X1 platform runs in the third Cloud that supports v6 because of us. We have our own internal Cloud as well and the choice was there by directionally where we said we need v6 support and we were prepared for those folks to say no and for us to take our business some place else. That is a choice that we both have to make. I think they chose wisely, right. So we had v6 support for the stuff we need. I think the public, the rest, will see that start to manifest itself but to be perfectly honest with you, that is not uncommon for us, we have done that on so many different occasions whether load balancers for DNS platform or whatever, it's not uncommon for us to be in that situation, and we are prepared to do the hard work, if it means we have to move our business, then we will. So I think long story short, yes it's happening, is it happening as fast as we would like for it to? No, but then again I am a very impatient individual to begin with. So yeah, I think it's coming, right.
JAN ZORZ: The queue now is right wing and middle front and then it's Anan and then it's middle front again.
AUDIENCE SPEAKER: Peter Hessler from Hostserver GmbH. In your slides talking about the IPv4 V IPv6 utilisation do you have any information about services that are dual hosted on both R4 and v6 and customers that have hosted v4 and v6, do you have an understanding of how much ‑‑ of when they choose IPv6 versus IPv4 and how much traffic could be moved over between the different protocols?
JOHN JASON BRZOZOWSKI: My services or off other people's services?
AUDIENCE SPEAKER: Either. Primary I am thinking of other people's services where it's ‑‑ there is both v4 and v6 path and which one is chosen and why.
JOHN JASON BRZOZOWSKI: That is a great question and a long answer but I do want to comment, and here is the thing about that one that is very interesting in and of itself, one operating systems, my mac, for example, and I expect this will get fixed in not‑too‑distant future, to prefer the querying of DNS over v4 today even it it gets a v6 address. There are other things Apple has fixed over the years, send query first, when service is on for v6 and a broadband house has v6 we are seeing that other than the performance aside, we are seeing like it's a pretty ‑‑ it's a significant shift but it's not like a flash shift overnight so I think if want to dive deeper into that I would be happy to work with you and others to bring back a dedicated discussion around what we see with our own stuff as well as what we see off net, I think we will have to be respectful of what the off net people are but we do see the pick‑up there quite a bit. If you are wondering, once you turn on v6 is there ‑‑ is it dead and not used? It's not true. It's actually ‑‑ we see cases where it's heavily used, 40, 50, 60% in a house that has v6 enabled, it's significant, right. Now, I am green so 40% is not enough, I wonder why is it not 80. But it's out there.
JAN ZORZ: I am closing the queues, whatever is on the mics can have a question. We are behind time so please, be brief, thank you.
AUDIENCE SPEAKER: Hello, Kostas Zorbadelos OTE, Greece. You had this slide about IPv6 deployment, IPv4 deployment over IPv6 transport for the broadband customers. And it is a difficult problem. Have you any evaluation, you mentioned about MAP there, is MAP your choice or are you evaluating anything or are you still considering stuff?
JOHN JASON BRZOZOWSKI: We think ‑‑ I mean, the fat lady still hasn't sung yet but anything is possible we are leaning towards some version of MAP, probably E and one of the things to take note of is the reason why this conversation is happening now is because if you think about the comment that I made about v6 utilisation growing, v6 is growing, v4 is declining, right, or it will eventually start to decline, so from our point of view it's the right time as an industry to have start having that conversation and carrying v4 over IPv6.
AUDIENCE SPEAKER: So you are still in conversation stage?
JOHN JASON BRZOZOWSKI: We see something like this happening next year.
AUDIENCE SPEAKER: Alain Durand, ICANN. First, congratulations on what you have achieved, I think it is really significant.
JOHN JASON BRZOZOWSKI: Thank you.
AUDIENCE SPEAKER: Second ‑‑ I have two questions. First one; do you have a place where you publically publish statistics about what is a bandwidth that is used over IPv6?
JOHN JASON BRZOZOWSKI: Not yet but it's something we have talked about amongst some of the measurement group. We think, we think it's important to share that, we are working on a place to go ahead and do that, Jason, myself and other people have been talking about formalising more of the measurement work that we are doing, stay tuned, we hope to be able to publish that automatically so people can consume that information on their own.
AUDIENCE SPEAKER: We are working with exchange points to do similar measurements and collect data for the network, that would be really, really useful. I would like to go back to the previous questions about why don't we see more bandwidth over v6, and you made the comment that to go to G or to F you see ‑‑ v4 and v6 why is it not more, because they have all services available on v6 and if users have all v6 available they should be like 80 something% which is the number of users you have passed?
JOHN JASON BRZOZOWSKI: I agree 100 percent with you. Part of the reason, I can point to two things and it was to Leslie, part A is, many of the content, the CDNs, the companies that are behind many of the large websites or the large volumes of websites haven't turned on v6 by default. We see kind ‑‑ we see the winds changing there and a lot of these players will turn on by default. If you look back, v6 launched, it's good enough for broadband providers to turn on by default, why not for the content providers. You still have things like Samsung TVs, players, they don't support v6 yet so I think the answer to your question is twofold: You see now if you look around the penetration for broadband is significant, right? So now it's to the point where we are starting to look around, doing the analysis and saying OK, well, content and consumer electronics really need to catch up a little bit.
AUDIENCE SPEAKER: It would be really good to have some kind of metric, measure of this particular factor.
AUDIENCE SPEAKER: Mathias Wolkert, Com Hem. I was curious, are you doing any multicast on v6?
JOHN JASON BRZOZOWSKI: We are not.
AUDIENCE SPEAKER: So you are not doing ITV multicasting.
JOHN JASON BRZOZOWSKI: We are looking to leverage IPv6 route to go facilitate the use of multicast without actually having to deploy a full‑blown multicast implementation. We have slides for that, I wasn't prepared to get into it but we can talk later and walk through it, the short answer is no we are not using what is defined today as PIM or what have you for multicast but we do have other plans.
AUDIENCE SPEAKER: Oliver from the RIPE NCC, we have three questions from the chat, actually. The first one is from Ian, FO Telecom, he says what about the major players like Twitter, where is the transition?
JOHN JASON BRZOZOWSKI: I don't use Twitter.
AUDIENCE SPEAKER: He is not here to gave response so go to the next one.
JOHN JASON BRZOZOWSKI: I don't think they are on but they are going to have to turn it on at some point so I mean, I am more worried about ‑‑ it's a valid question, I don't even in dodge it and I really don't use Twitter all that much, truth be told, I am worried about the Amazons, the Akamais, the guys who are moving huge volume of bits.
AUDIENCE SPEAKER: Thank you. The next one is from Sebastian from Norse Network, he asks how much v6 address space do you provide to broadband customers? What granularity of v6 firewalling do your CPE supports expose host forwarding, etc..
JOHN JASON BRZOZOWSKI: Default is 64 for residential, on or up to 60. Commerce gets 56, all over DOCSIS, and fibre customers get 48.
A. The firewall that we use for residential deployments is router based platform that varies by manufacturer but they all adhere to the cable AP specifications for e‑router.
AUDIENCE SPEAKER: From Mick O'Donovan BT Ireland. He asks: Have you ever thought of touring this talks to the boardrooms of dinosaur telcos by any chance, not that I know any dinosaur telcos.
JOHN JASON BRZOZOWSKI: I thought we had been doing that for the past ten years. Who else ‑‑ wow, I don't know. I mean, sincerely, we thought we have been doing that for the past ten years, we he has been talking to me to get me over here, I know this is not the boardroom but we have gone to many companies, offices, to have this conversation, so I mean, if ‑‑ I am opening to suggestions, let's put it that way. If you have a better idea, get it out there and let's talk about it but going to every one of them seems like that is going to be expensive travel bill for somebody, not me.
JAN ZORZ: OK, Jon, thank you very much, to coming over the pond and I will hand over to Shane, to introduce the next speaker.
SHANE KERR: Don't worry the next speaker is not me. So we have got I think kite interesting next presentation, Enno Rey is going to be talking to us about one of the dark sides of IPv6, multicast which may be enabled and may be causing risk that you are not even aware of.
ENNO REY: Thank you, Shane. Hi everybody, for my talk on MLD and certain properties of MLD, mostly security related ones. A very quick intro on myself. . I am kind of old school network security guy, I have started working provider space in the nineees, I have moved to enterprise space in 2000s and doing a lot of IPv6‑led stuff in the last ten years but it's like four or six years where I am only doing v6 projects in the interim. And the company I work for ERNW we are 50 people and we have a lot of research activities going on, we look at a lot of devices and protocols and technologies from a security perspective and this is where this kind of stuff was originated. And I will kind of present the results of research project that started two years ago at a presentation split into three parts, I will give a very quick intro on MLD and, say, certain properties of it which are of interest when it comes to a security evaluation.
Then, what we did actually to look at the security properties, how the lab was built and what type of test we performed and then we ‑‑ actually, the results, what we observed. And the background of all this research project is most of you will be aware when you look at the behaviour of IPv6 nodes, many of them in very early stage of their IPv6 stack initialisation process, emit MLD messages and you might ask what is this stuff? And when you start looking at what do the specs say about MLD, what is the relationship of MLD with IPv6 or what role does this protocol have, you might stumble across this one, RFC 4861 and neighbour discovery it says joining the solicited multicast node multicast address is done using multicast. I have a reputation of being a bit sceptical what is happening in the IETF, especially in IPv6 space and I think this is a very nice example ‑‑ OK the fonts are somewhat screwed but, regardless, this is a very nice example of, say, of a bad, say, expression of ‑‑ formulation in a class, what should be highlighted here is the is done, what does this is done say, is this a descriptive statement, describe reality, this is not what an RFC usually is supposed to do or is it like a prescriptive statement, it should done, if it was a prescriptive or normative it should have been in RFC 2119 terminology or capital should something. In any case, this can lead to a lot of confusion for implementers, and we will see there is a very unfortunate relationship between MLD and which can be attributed to this statement.
So, and that is another thing that was part of the roots of this ‑‑ research project, which was I gave a keynote at an IPv6 security event two years ago where I lamented on the complexity of IPv6 not contributing in a positive way to and in that I explicitly mentioned MLD. So, when you look at MLD, it turns out pretty much every stack has it and it has it enabled, it's there, you can't really get why or what is the role of this one, and as will very quickly turn out it has not so fortunate security properties.
Very quick run‑up what MLD does: In a multicast world, there is sender source and receivers of traffic, once this inter‑domain setting, so it goes across the Internet or routers. To perform the routing in a reasonably efficient way, the routers have to be in the locally adjacent networks, are there any interested listeners for multicast groups and they can express their interest, I want to get a certain type of multicast traffic and group traffic by means of MLD, for that purpose there is so‑called queriers and hosts, asking any of you guys interested in ‑‑ what multicast groups are you interested in receiving and the receivers are the listeners, they can send reports, like I want to get this or that types of traffic. MLD has two version, MLDv1 and MLDv2 which is slightly different for the mimics but we will skip this. And what is important for the, say, following part of the talk there is three ‑‑ in MLDv1 there is three types of messages, there is the query that the querier which usually the router sends out. There is reports where the listeners can report I have an interest in receiving this or that type. And once a listener leaves a group the system can send like a done message, I am no longer interested in this one. That is when it comes to MLDv1. And MLDv2 there is no done message any more, certain type of report message.
One particularly interesting type of conversation is the so‑called last listener call. As the querier does not keep state who exactly is interested in receiving a message, but only keeps state like OK there is at least one interested guy in the locally adjacent network, the querier has to, once it receives a done message, it has to find out is there anybody left and this happens by so‑called last listener query.
One additional thing that should be kept in mind is there is a feature on somewhat proprietary switches called snooping can learn behind which ports certain interested listeners are sitting and distribute the multicast traffic only there. I have to say, I am not a big fan of MLD snooping at all especially when it comes, well, to future large scale v6 networks, as this induces keeping state on switches which might not be the best idea in the world anyway, in future networks.
From the specs space, MLD has some very small built in like security precautions which can be broken down to an MLD message should have a link local source address, it should have a hop limit of one and a router alert option in a hot by HOP header. Messages not complying with these expectations are supposed to be dropped; in reality, as you can read in the research paper, which is I mentioned in the link section at the end of the presentation, this is not a case. There is many operating systems kind of violating these expectations by processing packets which do not conform to these ones.
Besides that, MLD doesn't have any inherent or built in security properties like authentication or integrity protection or something. It relies on the IPv6 trust model, which is everybody on the local link is supposed to be trustworthy, which might be a reasonable expectation or not.
In the RFCs, some more things we should keep in mind for what is going to come up in a few seconds: In the RFC itself, an MLD node must process a query that is to any of its interfaces, not like to multicast address but also to Unicast address, be it link local or global, which means it's possible to interact with MLD in a one‑to‑one manner, not over multicast groups and addressing but in a one‑to‑one manner with a device without other devices or nodes even noticing.
Secondly, once there is a querier in a network and another shows up which has a lower IPv6 address, that other device becomes the querier, so from like an attacker's perspective it's very easy to win an election, just show up with a lower IP address and you will become the querier.
Third, once there is MLDv1 and 2 as they are supposed to be interoperable, once an MLD 1 message shows up, devices are expected to downgrade to V1. Keep these things in mind. Some more things or aspects on implementation facts, on most common host operating systems it's MLD enabled by default. The messages are sent out very early, even before ND starts. And like reporting group memberships it's usually done at least twice, to cover the possibility that the message gets lost. Over all, different operating systems join different groups, which allows very easy fingerprinting, so you can just sit on a link and say easily get who else is there and which operating system this box is running. Which in the earlier days was considered to be security relevant, we do not consider it to be security relevant which is many ways of fingerprinting devices which is on the same segment you might be, which is on the actual security part of the research project, we look at three things: We looked at implementation problems, is there ‑‑ can we crash boxes by sending or how do they react if receive a lot of MLD messages or if they have huge messages? We looked at like do those boxes behave conforming to expectations lined out in the respective RFCs. Which in itself might not be too security relevant, but as will turn out once you look at subtle differences, they can be abused in certain ways and obviously we looked at like ‑‑ can we abuse MLD in just a way it is, even if boxes behave correctly according to the specs.
We had a certain lab, our lab is ‑‑ there is a lot of Cisco gear, which is why we use Cisco gear. Probably the results are not that different and from what I have seen in operational reality, things are even worse, but this is not ‑‑ all this is not window specific. What might be more interesting for some of you which tool sets did we use, and do we still use. There is IPv6 packet crafting tool of framework which is called K‑Run, which is written by one of the guys by Antonios, which is what I think the most powerful packet crafting tool in v6 space, and MLD capabilities in the course of had project and there is we use another thing called Dizzy, which is a framework that we developed on our own some years ago, which is allows stateful protocol once you come up with certain definition files and obviously we wrote those for MLD.
So what were the results of the lab testing that we did and all this is documented in much detail in that, one of the guys I mentioned, Jason, he wrote his thesis on this, so you can read much more in detail on this. It turns out, once you send huge MLD reports to network devices, and you don't even have to send them at a very high rate, it's more important that those packets are like huge and contain a number of MAR records and MLDv2 packets. The devices performance severely suffers forwarding packets get lost, control plane manage, management plane become heavily inflicted and that is not even ‑‑ that is not only apply to enterprise gear, when you pay close attention to we use Cisco 1900 which is not exactly the biggest gear in the world but it's the same for RSR1 K, devices, you can actually just by sending MLD packets at a certain rate and crafted in a certain way you can render RSR1 K pretty much unresponsive, which probably isn't best news in the world for some people here in the room.
Another thing is, it's very easy to perform amplification attacks on the local link only, this should be noted. By means of MLD. Just keeping in mind, sending a query triggers reports from every node which receives the query, usually ‑‑ depending on the specifics, in certain scenarios, namely MLDv1, nodes send reports for every group and send at least two reports, which usually means like if there is, say, Windows boxes, imagine like 200 Windows boxes in the same segment in a client VLAN, you will get by sending one report you will get 1,600 ‑‑ by sending one query you will get 1,600 reports back to be processed by the MLD querier on the link. There is some restrictions, you can't create many, there is some timers involved so admitted, it's not that easy to do that constantly high rate, but just the amplification factor should give you an idea that this is potentially bad stuff and again, there is no inherent security properties, so everybody can send these packets and generate a respective responses.
And looking at the third part, which was like can we abuse behaviour of MLD. When you look at like how Cisco handles and this is calling ‑‑ this is conforming to the specs, MLD packets, MLD packets can be sent to Unicast addresses and to both link local and global ones and to certain multicast groups, again this allows us to interact in a one‑to‑one manner, which means, say looking at the typical multicast deployment, it becomes quite easy to disrupt multicast traffic, actually there is two main approaches; the first one would be take over the querier role, which is easy, just show up with MLD reports sent from a queer ‑‑ lower IP address, send spoofed reports to remove multicast group from the device performing the forwarding. The querier doesn't have to be the device performing the actual forwarding so you become the querier first and then you stop all your kind of indicate, oh, wait, the router, you shouldn't ‑‑ oh, I'm done, so this one will send out the last ‑‑ oh, will be tempted to send out the last listener query but once you send it on your own, the legitimate forwarding device, which was the earlier querier, won't send out the last listener query, but will just have to multicast group removed from the forwarding table. The second one is slightly different, in that case MLD reports are sent to the Unicast addresses of the specific nodes, so you can say disable or deactivate the reception of multicast traffic on individual nodes, even.
We have like, I mean, a typical text scenario would be like once a shareholder meeting is multicasted, which happens in some of our customer organisations, you could either on a segment basis or on individual host basis, stop certain people from receiving this. By colleague Jason, he was a student who wrote the thesis, put up ‑‑ has prepared a nice video showing the exact way of the sequence and the commands to be used, which you can watch on your own, given I can't perform it here.
Which brings us to the question, OK, if there is ‑‑ there is potential for abuse, to state it cautiously. What could be done like from an operator's perspective, from a Sysadmin perspective, the first and foremost thing we recommend to do is, block MLD on a switch port level. There is no need, like, you have on your switches in a data centre, probably, you have the stuff like root guard, storm control, all this stuff anyway, just in that section like, why should we ever receive MLD queries from a given system at a given switch port? The problem is, there is no MLD guard, right now, so you would have to do this by port based ACL which is from operation's perspective is probably not what you want. Another thing could be like rate limiting. Processing of MLD or even disable MLD on a layer 3 device if you don't need it, which is what we have seen in a number of customer networks now that we have talked to and recommended to do. This doesn't break anything, it just reduces the potential ‑‑ heavily reduces the potential for harm. There is no things that can be done with MLD snoop enabled but what might be more important is modify the specs in a certain way. There is no need to receive or process MLD messages on Unicast addresses. There is no need like to accept things for hop limiter 1 and stuff like this, so overall we think some some adaptions should be done in the specification space, which is why Erik and I and Antonios came up with this draft, you can read what is in there. Actually, we suggest some tweaking for the relevant RFCs and we suggest a feature called MLD guard which would just protect on a switchboard level.
When it comes to CPEs, there is not much ‑‑ like we had a look at some CPEs in our home networks, they don't send out MLD queries so not much potential for harm there and home networks you don't have the scale anyway. Still, this might be a thing to keep in mind for some of you. Overall, to conclude, the thing is, you have to expect, once you enable IPv6, and let me be clear on this, I'm fully aligned with Jason here, we strongly recommend might be the wrong term, but we think IPv6 is the future and probably I don't have to tell anybody here in the room. Still, once you do this, be aware there is the thing, MLD, which comes on the back of IPv6, which in 90% of the networks, probably represented here, you don't need this. It was funny to see that even Jason ‑‑ John just mentioned, oh we don't use ‑‑ do multi‑stakeholdery cast over v6 but it is there and it has a large potential for harm (multicast) it's very easy in a costing environment for one hosting box to bring you down top of your X switch or layer 3 device, whatever those might be, both in watchful and physical settings, so keep these in mind. So taking care of MLD is part of what we think a proper network security.
Thank you for listening. Any questions?
SHANE KERR: Some questions from people who may know a thing or two about IPv6.
JEN LINKOVA: I have a few comments. I think it's every protocol have ‑‑ might have some security issues, right? And since that this protocol can over all the CPE on your device, it's kind of apply to every single protocol, right, you should think about protecting your router from individual hosting sending too many packets, it might be any protocol, it might be BGP, ICMP, it's more problem with doing proper plane control protection on your infrastructure, you could not blame IPv6 for no not protecting your devices.
Secondly, I am a bit concerned about filtering MLD on switch ports because you have MLD snooping enable you might break detection in this case.
ENNO REY: I agree with the first part, which is I mean, control protection might apply to a number of protocols and you should do this as a basic security precaution anyway, the thing is, Jen, the particular property of MLD here is, you don't need this. The ‑‑ what I found most like astonishing in this, during this research project, this is protocol that from a production perspective, you don't need, practically nobody ‑‑ I would say, Eileen outside the window, I am not sure if it can translate it in that way literally, 95% of the networks represented here in the room have no need for MLD at all, and still it's enabled. So it's not about something like well, any protocol can break your router but you have a protocol here that is enabled due to say unfortunate wording in an RFC and you just don't need this crap. And can break your networks. That is my point.
JEN LINKOVA: I would disagree that such a high percentage of network do not need it because it's drastically reduces amount of noise in the network, especially in the wireless case.
SHANE KERR: OK. I think after this we will close the mic.
BENEDIKT STOCKEBRAND: A couple of things. First one, when it comes to securing Layer 2, it's pretty much hopeless so what you should normally consider is going towards micro segmentation which will solve a lot of problems once you get rid of IPv4 in the network which makes it difficult to do this in a dual stack network. Second question is, how does it all compare to IPv4 IGMP? Because I mean, we have pretty much the same mechanisms with IPv4, the biggest difference really is that with IPv6, multicast routing becomes more practical, with all the inherent limitations which also leads to some of the vulnerabilities you mentioned. So it's ‑‑ yes we have a number of problems and in certain environments we really want to know what we are doing with this because it is kind of dangerous but it's not like it's major step back from IPv4 despite the fact it might sound like that, according to my opinion.
ENNO REY: The main difference when it comes to IPv4 is, with IPv4 IGMP you didn't need IGMP for most networks and types of operations, and it was enabled by default, if I would ask like the people here in the room how many of you have an IGMP daemon, component running on your Linux systems in your hosting environment I wouldn't seen many hands. If I asked the same question how many of you have MLD running on your Linux boxes and hosting environments, thinking about it pretty much all hands would go up. So that's the main difference. You have this thing anyway by default.
AUDIENCE SPEAKER: I agree on that because basically, if I understand you correctly what you say is, it's something where you have to change our default settings or operational procedures, whatever, to actually deal in these environments and set these things up accordingly.
ENNO REY: Somewhat. The problem is given this unfortunate relationship with ND, on Linux you can block MLD, you can't really disable it by type of SIS control or something, you can fit it which operations‑wise might be nice or not. You don't break anything. With Windows, if you like block MLD on local level, you break MLD, so with Windows you can't even disable it.
AUDIENCE SPEAKER: That is more of implementation issue rather than specs themselves.
ENNO REY: Well, that depends on your perspective.
AUDIENCE SPEAKER: Erik ‑‑ it's a comment on Jen not comments to Enno. Regarding why MLD is not like any other protocol, MLD has got some big bucks in it, for instance being able to send it with time to lifting 1 and not enforcing 255, that is a bug into the design. So we need to fix this. And the other stuff as well. And the other point about MLD snooping being important for reducing the amount of multicasting network in v6, actually it's not so true because at least one /SRERPBD, my employer, right, we do not implement on purpose (vendor) MLD snooping for link local multicast because we do fear missing that in and we do fear the amount of stakes and MLD snooping KSK disabled because it's not implemented in our gears. I cannot speak for others, of course.
SHANE KERR: All right. Thank you.
(Applause)
JAN ZORZ: So your next speaker is Vaibhav Bajpai, from Jaccobs University in Bremen.
VAIBHAV BAJPAI: Hello. It's too bright. And I am PhD student, I am supervised by professor. Today I am going to talk about measuring web similarity from dual stacked hosts and worked with Stevie and Sam based in London.
So the plot that you see is showing the evolution of AAAA websites over time, so it shows you how many websites are dual stacked today and what you can see is around 6% of one million dual stacks so provide which is around 60,000 websites and recent work which is 1, 2, 3, has compared performance of these websites during different times slices on this plot. But had there has been no study measuring similarity of v4 over v6, we want to know how similar are web pages over v6 to their v4 counterparts and what factors contribute to the dissimilarity, if there is one. Let be me very clear: When I am measuring similarity over time, I want to say; there is a browser running on our v4 only network and it's trying to reach a dual stack website over v4, I want to see how it looks and I want to compare the same thing off a browser running on v6 only network which goes to a website over v6, which v6 only I mean pure, not NAT 64 translation. So that is what you want to see, how similar are web pages over v4 and v6.
OK. So this is all the contributions that we provide. So, we show you a tool which will allow to you do this similarity comparison, and we saw that around 27% of ALEXA top have some sort of failure over IPv6 so there are some web page elements that failed to load over v6, by web page element I mean any embedded object like javascript images, something fails. And these failure rates are largely due to missing AAAA entries in DNS. On images. And these objects are failing both from cross origin and same origin sources, we will talk about that.
This is the first and this is all the first which is going beyond the route web page so if you look at most of the stuff is measuring the route web page but they don't go beyond the route web page of a website.
So how do we do this: So there are two well‑known page complexity metrics from literature, 4 and 5. I am not going to talk about this in detail, so this is not part of the presentation but this was just for completeness.
We measure against ALEXA 100 dual stacked websites, so these are some sample ones that we look at and this is the set‑up, so the test is SIM web and deployed on a Samknows probes, probes hosted by people at home for doing broadband measurements. So such kind of test is deploying on these probes and it runs twice, once for each address family, once for v4 and v6 and repeats every hour.
And we deployed this test on probes and gave it to people and this is the trial Kistelekiing of 80 dual stacked Samknows probes. Most of these are probes are connected in residential settings and this is thanks to you for actually hosting these probes for me so there are multiple people sitting in the room hosting these probes for me, thanks.
Let's look at some results.
So the realtime that I am going to show you were from ‑‑ collected for two months, starting from April 2015 to June 2015. So the first thing we look at is success rate, and the plot, the let's look at the plot first. What you see is 27% of ALEXA top 100 websites have some fraction of web page elements that failover v6, they don't failover 4 but only over v6. What are these 27 websites. Now look at the table, these are the 27 websites which have some sort of failure and the table will show you what percentage of failures you see. I highlighted some of the websites in red which I consider popular but this is from my own vantage point, and there is a horizontal line splitting showing you top 9% of the websites have more than 50% web page elements over v6 so half of the content you don't get if you are sitting on a v6 network with no translation. 6% of the website have complete failure so you won't get anything. Yes. And surprisingly, if you look at the last column, the ticks are on websites which promised ‑‑ they participated on the whole v6 launch day and promised to turn on v6 for their services.
OK. So, going back. Let's look at the top 6% websites which completely failover v6. So this is the time series starting from 2013 to 2016 of those which completely fail, and I don't know if you see there is a vertical marker on each plot showing when they started providing v6 services. So the bottom line here is, anybody who is doing IPv6 measurements should note that metrics that measure v6 adoption should account for changes in v6 readiness so, if something is giving you AAAA today don't assume it will give you tomorrow as well. Continuous monitoring is very see essential. OK. So, just remember the table. Let's look at the rest of the websites, which has some partial failure, OK. So, the first question we asked was. Where. In the network does a failure occur? And mostly due to missing AAAA entries in DNS, these are which don't have AAAA and this is mostly for elements such as images, JavaScript, JSON and CSS. How will it look over v6 only, so it's kind of surprising.
Both same and cross origin sources contribute to the failure of web page elements and we will talk about this in more detail. So by same origin source I mean let's take an example in this plot, Yahoo.com, so that has 28% of web page elements failing over v6 and I want to see how many of these web page elements come from so the same origin source because Yahoo can then control it so Yahoo knows that they are responsible for providing those web page elements. So how many of these web page elements are same origin? Look on the table on the right, it's around 83% of the failing elements are coming from same origin so Yahoo can do something about it, yes. If you look at the top of the horizontal line you will see 12% of these websites have more than 50% web page elements that belong to same origin sources and failover v6. You can roll that and turn on AAAA entries for those.
Then we flip the story. What about cross origin sources, so what are these cross origin content web page elements that fail? And it's largely tracking of websites, analytics websites that are actually failing over v6 so it's actually good news for you.
We also looked at what is the most common cross origin source which is failing across multiple websites, so double click .net, I think it's attracting third party tracking website, it spans across five websites in and ALEXA top 100 that is not providing AAAA entries. There is creative comments which is ‑ spanning through three websites and contributing to 76% of failures to three websites.
What is the take away from this message: Metrics that measure for v6 adoption should account for changes in v6 readiness, so we already looked about it, so continuous monitoring is essential to make sure that we are going upwards. Measuring only towards the route web page overestimatemation of v6 adoption numbers. So it's completely unclear to me whether these have partial failure can be deemed IPv6 ready, can they? And there are some few cross origin sources that once they enable v6 it will help a large number of websites so creative comments, if they start providing AAAA entries will be provided from T and that is all about from my end so thank you and I am happy to take questions.
JAN ZORZ: I think it's Martin first.
MARTIN LEVY: From CloudFlare, simply a statement. This is both depressing and brilliant work. Thank you.
JAN ZORZ: I think you.
AUDIENCE SPEAKER: Peter Hessler from Hostserver. Thank you for your presentation, you basically answered the question I asked during the Comcast presentation of how do you know, and I am very glad to see that you are looking not just at the index HTML page but the content of it, so thanks very much.
AUDIENCE SPEAKER: John Brzozowski, we met by e‑mail. One of the things that this talk makes me think of, we have been running some measurements like this for a few years, I didn't have the latest data to answer the other gentleman but it's important to track this stuff over a longer period of time and trying to ‑‑ one of the other things that I would encourage to you consider is take a look at the serving network, the owner of the domain may know nothing about IPv6 but the serving network might. And then extend what you are calling this IPv6 index to the serving network and say if this turned on v6 by default how much of that page becomes a v6 only experience.
VAIBHAV BAJPAI: I think there were two questions in that, comment. So first of all, collecting the trend so the test is actually still running so I have a long data set so most likely things might have improved over time but maybe I can show it later, somehow. The second was looking at which networks are responsible for the failure. So it's not clear to me, when the test is running and fails, you don't see the IP end point so you can't know from which network might be responsible. You can correlate it with v4 and then try to assume but then it's an assumption, so is it OK? If it's OK then it's possible. /STPWHRORZ Zorz it's the back mike.
EMILE ABEN: Really interesting work. Are you making the data, the detailed data about what fails publically available so people can actually work on fixing this stuff?
VAIBHAV BAJPAI: Yes, so we have a paper on the submission but I can release the date early as well.
EMILE ABEN: Excellent.
AUDIENCE SPEAKER: Patrik Wallst, IIS. I think there are more dimensions of this that you need to consider as well, for example TLS, I have observed several times where one website is available over TLS and not over IPv4 or IPv6.
VAIBHAV BAJPAI: Yes, this is only measuring ‑‑ that is not doing TLS. But if you want to know how many websites failover TLS, look at the toy that I developed, it's on the IPv6 Ops mailing list and it is using the platform measuring against ALEXA authentic websites, over 443 and if your website is top ten dual stack most likely the measurement is failing so try that tool.
AUDIENCE SPEAKER: Both over IPv4 and IPv6?
VAIBHAV BAJPAI: Yes.
JAN ZORZ: OK, thank you, thank you, very much.
(Applause)
We understand we are standing between you and lunch but we have some housekeeping to do. First of all, please rate the talks, we would like to understand which presentations were more interesting to you and we would like to give 30 seconds to Leslie to mention something.
DANIEL KARRENBERG: Hi, in case anyone doesn't know about net girls, it is an organisation for all women in the networking industry and we are having our lunch today at the second floor lunch place just look for the net girl signs. Thanks.
JAN ZORZ: OK. With that, bon appetit.
