Archives

IPv6 Working Group session
26 May 2016
4 p.m.


JEN LINKOVA: Good afternoon, welcome to IPv6 Working Group. If you are interested in DNS ‑‑ stay here anyway. This room is bigger.

So, please take your seat, fasten your seat belts, turn off your electronic devices. Exits are over there.

Welcome. Agenda for today. Actually no administrative matters, fortunately, except for thanking our scribes and everyone who helps with the agenda, and agenda today is mostly about stories, or at least some very interesting stories and let's start to be on time because after this session it's going to be BoF by screening the net of right movie in the tutorial room. Everyone invited. Before you go to have a beer, or any other drinks, please rate the talks. Unfortunately, I cannot promise you any Amazon voucher or any other gifts. The only thing I can promise is we'll take your feedback into account when planning the agenda for the next session.

Our first talk today is about deployment in Latin American Caribbean region.

GUILLERMO CICILEO: Good afternoon. I am from LACNIC. LACNIC is one of the five RIRs. LACNIC is the RIR of the Latin American and Caribbean region and I'm here to talk to you about an IPv6 deployment study that we did in our region.

This was a study that we partnered with Banco Cav, it's a development bank for Latin America. And we did a survey and a lot of face‑to‑face meetings inside the regions with mainly with the ISPs, and with other organisations, to be able to understand why the region is delayed in IPv6 adoption, and to know how we can improve that situation.

This study was done during the second semester of last year and the results a published at that address, portal IPv6 dot LACNIC.net.

We have a big report with many different sections. I will only show you some of the main findings, but the study includes several parts that are, for instance, the results of the survey that we did among our members. The face‑to‑face interviews with the operators and with other organisations like academic organisations and Government. The study includes different points of view, as I said and there is an economic model about IPv6 deployment for helping the ISPs that want to deploy IPv6 and there are a lot of guides and recommendations and lest practices in this study.

So. I will show you only the main results. One of the first activity we did was to define an indicator, an IPv6 deployment indicator for the region. We raised that on several partial indicators that you, I assume you all know, like the ones defined by Google, APNIC, Cisco and others, but we adapted that to our region because we gave more weight to the planning status, it allows us to compare the different countries in our region that are still in a planning stage of IPv6 deployment.

The partial indicators in which this index is based are the usual ones, you already know, like presentation of IPv6 prefixes with observed traffic with respect to the allocated IPv6 prefixes.
The presentation of aut‑num systems that have IPv6 transit.
The content indicator which is a weighted percentage of IPv6 enabled sites. And also the percentage of IPv6 capable end users.

Based on that we defined this index that gives ‑‑ it's similar to the Cisco formula, but it gives more weight to some of the planning status, that we consider the planning status, like the presentation of IPv6 prefix with observed traffic and the transit ASNs. And it gives us 30% of weight for that and a 70% for the content by users.

With this indicator, only a country can only get over 30% if that country implements some form of IPv6 deployment to the end user.

This gives us a tool for quick comparison between countries. We have here in this graph a comparison between the Latin American countries and the main economies in the world, like ‑‑ well you have there, from Argentina to Venezuela, who is here, which is here, and here some of their reference countries like United States, Belgium and others. And as you can see, there are at least two countries that you can think that they are on the same levels as the more evolved economies, those countries are Peru and Equador, but the other, the rest of our countries are below that index.
Well I think no one can see those numbers here, but this table shows the partial indicators, and then you can see that the problem is mainly here, which is the presentation of IPv6 capable users. In that column there are only now five countries that have more than 1% of IPv6 capable users. That is the number of IPv6 enabled users in their homes for instance. And so, you can watch that column and there are only, the main countries like I mentioned before, Equador, Peru, and there's also Brazil, which is interesting, and Bolivia. In the last month, Trinidad and Tobago.

With respect to the content, this indicator shows that all the countries have more or less the same content of IPv6, about 50% of IPv6 sites enabled, but this has to do with the fact that in every country, we go to the same sites as in all over the world. So, most of the traffic goes to the sites, the global sites that already have IPv6, like Google, Facebook and CDN, content and NetFlix, and so ‑‑ but what this shows is that if the ISPs implement IPv6 now, at least 50% of their traffic will go native on IPv6 with decreased load an the boxes.

As part of the study, we did a face‑to‑face visits to some countries. We had a list of ten countries. We were there for face‑to‑face meetings in the countries were Argentina, chilley, Bolivia, Peru, Equador, Columbia, Venezuela, Panama, domain anyone ken republic and Trinidad and Tobago. We say face‑to‑face with main ISPs but also with academic networks and governments.

We identified at least three successful cases at that time. And the time of the study with which is a state company of telecommunications in he can door, contake owe, that is a cooperation in Bolivia. And Telefonica Peru, that is the first ISP that deployed IPv6 in the region.

And during the study, Brazil was not part of the study but within that, Brazil statistics started to grow, and IPv6 up take was increasing steadily. As you can see in that graph. There are two main operators in Brazil that are operating IPv6, they are Claro net and VIVO GVT. And one important thing in Brazil is that there are not only ISPs deploying IPv6, but also big content providers like Globo and UOL that have adopted and deployed IPv6.

There are two other countries that this year began to do some experiment on IPv6. Trinidad and Tobago is growing, it's still starting to deploy IPv6, but as you can see in the second graph, the traffic is growing, and the other one is Belize, but Belize is more has more variations in the traffic. And the numbers are still very low.

Some of the main findings on this study. First of all, most of the ISPs are not ‑‑ are still not offering IPv6 to end users. But most of them have deployed IPv6 in their network core. The transition strategy adopted is dual stack with native IPv6 and CGN for the IPv4 addresses. Almost no one is expecting to deploy other transition strategies like NAT 64 or 464 XLat, map, they don't even considering to do that in data centres, there are some operators that are deploying new data centres in the region, but they are not using neither of these transition strategies.



The fact is that the countries that have more evolved Internet, that have large Internet penetration, are the ones that are most delayed in IPv6 up take, because of the lower growth rate in their user base. So, many of them said that they have still IPv4 addresses available for some years.

We find a lot of problems that the ISPs mentioned to us. Mostly of the problems came from the equipment they deploy for the end user. For instance, the CPEs that are supposed to be IPv6 enabled but when they deploy the equipment, they have problems. The equipment doesn't comply with some of the IPv6 needs.

One of the ‑‑ and that implies that the operator had to do a long test of equipment. And that is a delay they have for deploying IPv6 because they need to do test CPEs for a month, because they have different equipment. What was mentioned by almost all operators was the provisions systems. They said that the provisioning system don't support IPv6 so they sometimes have the equipment, the CPEs and the backbone equipments that are supporting IPv6 and they can deploy IPv6 to some officers but not in an automated way, so they are still waiting for their software system to support IPv6. And also, the business systems, some of them don't support IPv6.

With respect to training of the operations or help desk, they mentioned that as a problem also, but in most cases, they can solve that by internal training. In the universities and research networks, we found that most of the universities have deployed IPv6 to the CPE, but internally they have some problems, for instance, like firewalls or Wi‑Fi equipment that is not configured for IPv6, or some equipment doesn't support but mostly, they are not taking into account IPv6, for instance, in the wireless networks.

And with respect to governments, there are a lot of governments implementing plans of eGov systems or portals and also community Wi‑Fi networks that are not taking into account IPv6.

Finally, we are working on an economic model for helping the ISPs to evaluate the different alternatives. And we decided to do a comparison between three alternatives, which are deploying IPv6 with dual stack with Carrier grade NAT which is most of the ISPs in the region are working. Comparing that alternative with doing nothing and buying IPv4 addresses in secondary market. The third alternative is also deploying Carrier‑Grade NAT and not deploying IPv6. And we compare that using the net present value of the file of the next five years for the operators, there are a lot of parameters there that an operator can adjust, and this model is a high level in our page for the operators to find and to model their own business.

As I said before, the complete study has been chapters and is available for downloading, it's translated to English. It's address portal, IPv6 dot LACNIC.net. There are links to the different sections of the study, and the programmable interface of the economic model, and we hope to be publishing the information in the future in the following month.

That's all I have to say.

Thank you and if you have any questions, I can answer them.

AUDIENCE SPEAKER: Hello. Jan Zorz. I have a question, something really sort of like jumped out. So, hen you said that operators, and there was explicitly said mobile operators, are not looking into deploying any sort of form of NAT 64 or 464 XLat, it's usually used by mobile operators these days to deploy IPv6. Why is your continent different than the others?

GUILLERMO CICILEO: I don't know, but they are doing that. When we talk with the operators, they said that they want to go to the known technologies, and most of them already deployed Carrier‑Grade NAT of the address shortage, and we talked to several of them and asking why they are not deploying 464 XLat for the mobile. They are not considering that model. One of the things they are doing is deploying Carrier‑Grade NAT on the mobile and freeing up those other systems and using those for the fixed access, for the broadband fixed access, because they consider that for the mobile users, it's enough with Carrier‑Grade NAT and IPv4 and not deploying IPv6. But... that's a perception from them. We tried to change that, but that's what they are doing.

JAN ZORZ: This gives me an idea of maybe proposing for the next LACNIC talk how CGN v4 only has no exit strategy. And another thing that I would like to mention. You said that IPv6 CPEs are not really IPv6 ready. I would like to thank a lot to pick‑up section of LACNIC, they translated RIPE 554, that's the requirements for IPv6, for equipment into Spanish also, so you have Portuguese and Spanish versions, so you can distribute that and also suggest the operators to use it when they are buying the equipment. Thank you.

JEN LINKOVA: Any other questions? I have actually have one question. So, you mentioned that you graded this way for operators to work on the economics of various options, right, a spreadsheet of how you did it. So, it looks like for them IPv6 still looks like the cheapest option? You mentioned ‑‑ you have a slide with all this information ‑‑

GUILLERMO CICILEO: It is the cheapest option. It depends on the parameters of the solution they chose. If ‑‑ it depends on the grade, it depends on different parameters. There are in the model. Well, there are several parameters here that the cost of the different equipment and the ‑‑ for instance, the session cost and the replacement rate of the equipment and several options. But if ‑‑ with the numbers, we decided for the first instance of the model, IPv6 is the best strategy. If you have a very loose model growth rate. Maybe staying with IPv4 and buying other systems can be a solution now, but for short‑term. It depends also on the number of years you expect to be in your business and several parameters. But, comparing all of them, yes. And in the case of the first two options, which are deploying IPv6 in dual stack with CGN, but they are already doing. In the case of mobile ‑‑ we did two different comparisons there, one is including the CPEs and the other one does not include the cost of the CPEs. So in that option, in which you don't take account the CPE cost, it's more like the mobile operators is clearly more advantage to deploy IPv6. It's half of the cost almost.

JEN LINKOVA: Thank you. Any more questions. Then thank you very much for coming and sharing your experience.
(Applause)
Okay. Now we just have listened to why some people do not want to deploy v6 and now we can hear story about deploying v6 only.

LUUK HENDRIKS: Good afternoon everyone. I am luke, I am with the University of Twente in the Netherlands but this is something I have done at my own place in my own time. So this has nothing to do with stuff I do at university.

This is the notorious thingy. Yes it is.

All right, I really hope that I see a lot of hands now. Who recognises this thing? I this was the password for the then called IPv6 only experimental SSID at, what was it, I think up until one year ago, there was the NAT 64 Wi‑Fi network here on the RIPE meetings. And I think it was two years ago that I was at my first RIPE meeting in London and it was the first time I connected the to a v6 only network. I never thought about it that you could do this.

So, I was (v6) inspired you could say, and I asked myself, can I realise this kind of thing at my own place? Can I do this at home? Will it work? Or do I need a lot of expensive equipment? Now, I'm from the Netherlands so I don't really want to spend a single cent on extra equipment. But I do want to get this thing working, right. So, can I do this with the stuff I already have? Can I do this in a way that's not too dirty?

What I would like, is that I keep my gateway, my home router in a state that is, say, quite vanilla. I don't want it with a firmware upgrade I break or brick my entire gateway. If I achieve all of this, I think I would have a solution that's applicable and deployable for you know, a default home setup.

So, let's see if we can get that in working order. We had a presentation from Jen yesterday explaining how all this DNS 64, NAT 64 stuff works, so, I won't bother you with that stuff again. But just to say a very quick recap. This could be a home situation. Right. You have your v6 only client asking for an AAAA of something that doesn't have an actual AAAA records. Your gateway, running DNS 64 will generate a AAAA record for with you a prefix, gives it back to you and you think okay, that's a normal v6 host. Then you start sending traffic again to the same gateway, also running NAT 64. The gateway knows what to do, it translate the stuff into IPv4 and it goes out off your network and everything is just fine and dandy.

Now, I want to keep my gateway vanilla, right. My gateway doesn't really run ‑‑ it doesn't have the capabilities to run DNS 64 or NAT 64 by default, so I need another device in my network, if I have one, and I have one, to do this for me. So I can keep my gateway in a stable state, in a safe state, and I just run all of the new stuff, for me the new stuff, on my home server which I already had so I didn't have to buy one, so that might be called cheating, if you look at the requirements, but okay.

So, what happens then? I run DNS 64 on my home server. I ask my home server for an AAAA records of which, there is no really existing AAAA records. It does the translation, or the syntisisation for me, and then my laptop, my client, just continues to work like it's a regular v6 connection. My traffic goes through the gateway, the gateway has a route saying hey, this is the NAT 64 prefix, I have to translate it, well it doesn't know it has to translate but it has to be translated and I also do it on the server. So it goes again to the server, right. The server knows hey, this is a NAT 64 prefix, I will translate it to v4 and it will send it back to the gateway.

Now, I have to admit when I was making this diagram, that I felt a bit dirty. So, please bear with me, don't run for a drain cleaner or battery acid, or what was it again? Well, we wanted to keep the gateway in vanilla state so we have to make some dirty solution for that, I think.

So, if I don't want to buy new hardware, I need software to do the stuff. You have plenty of options. I have a slide with some options later on. But I chose to use power DNS with a lieu a script that does the generation of the addresses for you. And for the NAT 64 I choose Tayga. Actually I tried EC Decis, but I couldn't get to to work and I couldn't get in touch with the developers, I'm not sure if it's still actively.maintained or not but I went for Tayga, and an important note here is and we'll see why because Tayga is a user space NAT 64 implementation.
So start the configuration. It's really not that much work. It's literally ten lines of configuration spread over two files. So that's doable I would say. The upper part here is the Lua script for power DNS. This is from the actual official documentation, so this is not a Hackey thing that I hacked together. This is really a feature from power DNS.

The bottom part is the Tayga configuration, that's a bit more exciting maybe, because what Tayga does is it creates a tunnel interface on my home server. That's done in NAT 64. It will translate v6 with that prefix into a v4 address. But it's important that this v4 address, it's not leaving the server at all. You don't want to see that 10.64.0 range anywhere in your network. This is really only on the server. Then it should leave the home server and reach the gateway.

So, lines of configure plus a few other things the again the word NAT, it gets dirtier and dirtier with every slide. So it's able to go on to my v4 network, right, that's 192.168 etc. And you need to enable forwarding. Please do not think that this is a let's say 100% good guideline to follow when you try to do this, because this is obviously gone after I reboot, right. Then on the gateway, we need a route to get this NAT 64 prefix to the server, right, because my laptop, my v6 only client is sending out traffic to this generated quad a records with the 64 prefix and my gateway says hey that has to go to my home server first, translate it to v4 and get back. Right.

And then, finally you have a v6 only network at home. That's what I thought. But, there are still some missing pieces to get a, let's say, bearable user experience. I didn't think of a very basic thing. And then stuff doesn't work apparently. And it's the DNS. What's in your resolver.conf. Maybe there is stuff left in there and everything worked fine and dandy. What is there in the search field and domain field, nobody knows, it's probably not even there, right, so we need to fix this and we have multiple options there. I guess you are all familiar at least a bit with multiple flavours of DHCP in v6, I set up the version of DHCP which does not give out addresses but only the additional configuration stuff so the name servers and which domain to search in. Then I found out that I could actually change this on my gateway to put it in the route advertisements, which is nicer I think, and that's done with rDNS and DNS S L. It's important to mention that this is stuff you do on your gateway. One of the assumptions when you start out with this is that you are already dual stacked and most people in their home will have, I think, a slack out of config setup. The route advertisements already came from your home gateway. This is nothing to do with the home server, that's only there for NAT 64 and DNS 64. Right. So, it really depends on what kind of device you are running here. If you have some CPE from an operator that doesn't allow to you change anything related to v6, well this will be a problem. Under all of the graphical user interfaces, there is probably one of these things running anyway, so if you manage to go to a custom settings field or whatever in a degree of your router or you can access the actual configuration files of these pieces of software running on that router, then you are able to change these things. Now, I'll leave it up to you if this is vanilla or not vanilla any more. But, it's needed to get a bearable user experience.

So then, now you have a v6 only network with DNS right, that's really nice. Apologies if this means something dirty in Danish, I don't know, I didn't check.

I want to connect to my home server because I need to check if everything is okay. I tried to ping it. And it doesn't work. I did fix the DNS, right, so what can it be? Well, I need some AAAA records for the host in my network. I didn't need those things before, right. When I was dual stacked. I could always connect to them. How did that work? Well, I guess like many of you, I used static D PCP leases are label or name or whatever thinks then up ended to the E TC resolve for a thing running D DNS mask or whatever and there is some Mac I think there and everything is solved for you and you never think about it. If you do a v6 only thing. In my case I had to do this myself right, I was already running power DNS to do NAT 64 so I just created a local zone with AAAA records for my local machines.

Okay. Then it really worked, then I really got a similar experience as I had during the RIPE meetings, right, which was for me sufficient. Very good actually.

So, I should evaluate it, right, I should measure if things are okay. And, I want some feedback from my user pool whether it is indeed offering a good quality of experience.

So, measuring performance, it's not too academic, I start iPerv and I connect to my machine at the university. Let's see how that goes before we get to the results we start Htop and we can see that there is a certain process that's running at 99.3% CPU, which is more than enough. And coincidentally that is Tayga, the NAT 64 implementation. As I said this is a user space application. And that's nice, because it is probably installable, executable on a large list of devices if you can get it compiled on there. It's not so nice if you get this running on a smaller device running open WRT or whatever, and it really burns out all the resources in the device. Right. This is an Intel I 3 server as you can see, this is not that problematic, but it gives a nasty feeling if you know there is a device running at 100% CPU within your network.

The good thing is that it didn't really affect the performance in terms of network throughput. As you can see, I hope you can see, it almost reached gigabit speed, which it should, yes, because we have gigabits in that part of the Netherlands. And this was, like I said, a measurement to the computer, my computer at the university that entire building is behind a gigabit switch still and this was done during office hours and there is always a few Professors streaming, videos or whatever, so I think that number at that time shows that it's actually performing in terms of throughput, it's performing fairly okay.

Still, I don't like processes running at 100%. So, I really think about this if you want to deploy this.

Then, that was the measurement part. How do the other users in my network experience this, right? I'm a nerd, I like this stuff, I know where to look if things don't work, right, but it's really important that my network is functioning at 100% at my place, right. I will try to represent the entire user pool at my home, so you can see why it's so important. That's only my girlfriend. And well I won't have any financial loss if the network goes down, I might lose several other things. So, it's really important that things will work now and will work in the future, right. Again, I see why big companies are a bit afraid of going v6 only because of possible financial loss. But, with only a person, your significant other, in that network, it should be up and running as well.

So, were there any problems? Actually, not so much. There was one single application that didn't work out of the box on the v6 only network. And that's an instant messaging thingy and there was a tick box, and option that said do IPv6. And I think it said experimental or something. We ticked the thing and it worked. It was the only application that didn't work, right. And if you attended, what was it the last two meetings, you attended the v6 sessions, then you know that this was also the most common thing on the RIPE meetings. It's the applications that have hard coded v4 lit relevance in there or whatever, that do not use the D L S so they cannot leverage the AAAA records if things don't work.

There was another thing and I think that's even more interesting than this. If you have to choose (lit relevance) to choose which network to connect to, then you first have to know which ones are yours, right. Well my girlfriend those that this stuff is called Hoare as, Hoare as is a character from the best PC game that was ever made but you all know that of course so I won't go into that, and all my device resource named after things from the PC game. So she knows it's Hoare as something. I have to connect to Hoare as something. As you can see there is the normal Hoare as, there is a 5 gigahertz version, there is a guest network and there is the NAT 64, 5 gigahertz network. So which one do you choose? I asked her earlier this week, can you send me a screen‑shot from your SSID list from your Wi‑Fi, choose a Wi‑Fi network list. And I think we can learn two things from this. .the first thing is she has no include how to do Proxy‑Arp screen shots on a Mac book because she made this picture for fun. She wants to use the Internet to get on Facebook, right. There is this other thing, well Hoare as guest, that's the guest network, that's not the one you want, but what is this NAT 64, 5 gigahertz thing? That's probably something I don't need, right, so why would I connect to that? And if I don't say that this is the Proxy‑Arp network you can use this, she will always go for the simple SSID, because that's the normal Internet that I want.

And this is also a thing that we saw at the RIPE meetings. We had a big discussion on the mailing lists, can we rename this? Can we make this the official RIPE network? What if somebody breaks, can we offer support, etc., etc.? I think it was a very good step to go from IPv6 only experimental to actually RIPE MTG NAT 64 if I'm not mistaken. It was funny to see that the name of the SSID does matter how people interpret this. Can I use this network? There is a scary number in it, ill use the normal one.

So wrap‑up. This is really doable. This is really doable. You might not agree with everything that's on the slide and it's not always very elegant solution, but I would say try to improve this. I can do this so you can do this too. The good thing is, your home network is probably the easiest to try this out, right. I get that you don't want to try this on a Friday afternoon in a big network. Right. With these possible financial losses. But at home, there is not that much to lose.

The DNS is a big thing right and I don't want to get start with this DNSSEC discussion that was there yesterday. But there was another thing, if we have this and ‑‑ if you have a v6 only network at home and you are connecting to other hosts by DNS names, you don't really know whether you are connecting over Proxy‑Arp v6 or maybe v4. I noticed this when I connected to my machine at the university which has an A record but not a AAAA records for reasons.

So what happens it you connect with the v6 only network, connection will work, I do have proper v6 at home. I do have proper v6 at the university and in the end you are using NAT 64.

So, are we hiding the actual hiding IPv6 everywhere, maybe, my opinion is this is a thing to aid in a transition. And I think the most important thing is if you deploy this at your place, so think in v6, and this is the most important take away message, if you ask me. If you are lying on your couch programming or whatever on a v6 only network, you will never make the mistake of hard coding v4 lit relevance in a piece of code, because it won't work if you try to run it.

If you are configuring a remote service and you know, do it quick and easy and dirty and just use a v4 address because it's easier to remember and you are too lady to update the DNS, it won't work because you will not connect to the service because you are on a v6 only network.

So try this out. Here is a list of other options. Like I said I used power DNS and Tayga. I know that Jool is a kernel module NAT 64 implementation. I'm currently trying to get it up and running because this might improve the results of this iPerv plus had the deposit test and that's something that we want. Depending on your gateway or CPE this might be very hard or easy. I don't know. But what I want to say is try this at home. Please try this at home. At the very, very worst you will learn a thing or two, and I think that's always a good thing.

So, thanks a lot. If you have any questions or comments or improvements on this dirty set of, I would be very, very happy to hear those.

AUDIENCE SPEAKER: Andre: You probably had from from the youngest generation that don't have a smart TV at home because if you had a smart TV at home you would notice that I haven't yet any smart TV that would know anything but IPv4.

LUUK HENDRIKS: You're right. I don't have a smart TV, I I don't even even have a dumb TV which would also not have a v6 functionality. Yeah, I guess you're right. I don't know ‑‑

AUDIENCE SPEAKER: Would it be tested from those who don't have a wired IP phone.

LUUK HENDRIKS: I have one somewhere in the attic but it's not connected. You're right.

AUDIENCE SPEAKER: Because this is a doesn't work everything else work. For the record I haven't connected to the RIPE network yet during this meeting, I'm all the time from NAT 64 and haven't spotted any problem.

LUUK HENDRIKS: Here here.

BENEDIKT STOCKEBRAND: What kind of home server do you use sort of? The home server you run Tayga on, that's going to 99%, what sort of hardware is that?

LUUK HENDRIKS: Like I said it's an Intel i3 machine XX 8, 64 bit Debian 8.

BENEDIKT STOCKEBRAND: Okay. An external bandwidth, do you have to about a gigabit. Your bandwidth to the Internet from your home, is about a gigabit or did I get the numbers right?

LUUK HENDRIKS: It should be 1 gig bet symmetrical fibre.

BENEDIKT STOCKEBRAND: Okay. There is something you can do to improve things. I'm not much of a fan of NAT 64 but it is a fallback for a number of things, if you combine that with physics proxy whatever, it might actually improve, so, you have a proxy talking IPv6 so your client but IPv4 to the outside. In slightly bigger setups you have that in a D M set as an application of a gateway. That tends to solve quite a few problems. And it's not so much the literal addresses in the code. It depends on how address solution works. If you get host by name, on return IPv4 address, it should be get address info and otherwise using fixed address family IPv4 A F I net that's the real problem. It's not really a problem to fix as long as you have the source and know what you are doing, but those two can be two rather nasty problems.

AUDIENCE SPEAKER: Hi, so, let me be the devil's advocate here. I have a perfectly working dual stack network at home. What is my motivation to go to criminal my network by going NAT 64 with v6 only? One motivation I saw from your last slide is you can think in v6. I do think in v6. If I don't, I can always down off v4 locally in my machine and think in v6. My point is, just because something can be done, doesn't necessarily mean that it is a valuable thing to do. Thanks.

LUUK HENDRIKS: I agree. When I was making this presentation, I was trying to come up with a list of motivations, and to be honest, the most motivating thing for me was to be able to say I have a v6 only network at home, look at me I'm such a cool nerd. That's the thing. While I was doing it I learned a lot of things. I think that's a great motivation will as well. I also ‑‑ well, like to think that I think in v6, but Ien coward some problems with that by myself as well to be really honest. So for me it was worth all the effort. But your model may vary.

JEN LINKOVA: I have one quick comment. If you by any chance develop any application for Apple store, it might be a very good motivation for you to do this.

AUDIENCE SPEAKER: Dmitry most master. I think that NAT 64 is the only NAT I actually like and so I would like like to feel a nerd and have a similar thing. I have a question, have you considered running this thing over 2 nil v6, for people who don't have real v6 at home. I don't mean to sound weird like, for example if you run a tuna broker or something you want to use that v6, do you want to use the NAT 64 on that?

LUUK HENDRIKS: So your question is whether you can run this set if you have a v6 tunnel instead of native v6.

AUDIENCE SPEAKER: Most countries have ISPs that dont's do v6.

LUUK HENDRIKS: To be entirely corrective do have a tunnel at home. I don't have native v6 at home. So I'm running this over a tunnel providered by my own ISP.

AUDIENCE SPEAKER: Then you are a truly nerd, because you synthesise v4 on top ‑‑ okay, I got it. So ‑‑ and one more question. I bet your girlfriend doesn't really do VPNs but have you found any VPN provider

JEN LINKOVA: I know one, I use one.

AUDIENCE SPEAKER: Tell us what it is.

LUUK HENDRIKS: I just realised that we had problems once with her work laptop trying to get a VPN running, and I'm not sure whether she was on a NAT 64 network by then. We, indeed, don't use a lot of VPNs, so I'm not entirely comfortable with answering this.

AUDIENCE SPEAKER: Maybe ‑‑

LUUK HENDRIKS: It's a nice next thing to test, definitely. Yeah, thanks.

JEN LINKOVA: Last question because I am afraid I have to cut the line, we are slightly late.

AUDIENCE SPEAKER: More of a comment.

ELVIS VELEA: I connected to the NAT 64 network here, and my Skype and my drop box don't work.

JEN LINKOVA: Backdrop box well known issues, they do not ask for AAAA for some names, I reported it like two years ago, I got no reply, Skype I don't know, Google works.

LUUK HENDRIKS: I think this was quite well known the last two meetings as well. These are know forious problematic applications.

RANDY BUSH: Two things. Number 1, don't worry about drop box. The new drop box doesn't run in user space. It runs in kernel space. This will all be solved. I'm serious. Be scared.

Number 2 is, here we are in 2016, and we still can't run easily and painlessly a pure v6 network. This is disgusting.

LUUK HENDRIKS: I agree.

JEN LINKOVA: We are kind of slightly late. So, while ‑‑ I want to do some shameless plug, I'm going to make your life easier and your configuration much cleaner, come tomorrow at my lightning talk about IPv6 being so easiey.

WOLFGANG ZENKER: Good afternoon everyone. My name is Wolfgang, I am with punkt.de, it's a hosting application developments company in Germany. And I want to tell you something about accessing Open Source source code over IPv6 only matrix.

Well there is loads of definition of open source, but one thing basically everyone agrees on is Open Source means the source is openly available. But, how true is it if you are using IPv6 only? We tried to find out by looking at a huge collection of Open Source offerings, the FreeBSD ports collection. For those of you that don't know it yet, FreeBSD port is basically a big file that tells how to download, patch, build package and install a piece of software. The measurement was done in mid‑December. At the time we had about 25,000 Open Source projects having a port in the ports collection.

And the source code for these ports was hosted at about 5,000 different hosts.

Okay. Of those, well almost 4,000 had only an IPv4 address. Not a single one was on IPv6 only. Well, about 1,500 had dual stack. And a few were basically broken.

So, that tells us, of these 25,000 ports, about 40% were not fetchable if you were on IPv6 only. That looks pretty bad. But, unfortunately it gets worse. Because, ports might depend on our ports. And if you take that into account, about almost 70% of the ports we have in the collection could not be built if you have IPv6 only. Well, that's a situation I think that should not happen in 2016. Well we tried to find out why this situation is that bad. And we did that by looking at the top 25 hosts providing source code. These 25 hosts, and don't have IPv6 ‑‑ these 25 hosts provide source for about 8,000 ports. And thee of them hosting together almost 3,000 ports, don't support IPv6 because they use the CDN that does IPv4 only. In this case CDN was fastly, which looks weird because that CDN is pretty new, it started in 2011, and according to their API documentation, they do IPv6 on the content facing side of the CDN, but the consumer side, the front side, is IPv4 only, and I could not find any commitment on their website to do anything about it any time soon.

But at least the users of that CDN are aware of the situation. They see three sides are basically used by the Python community and by the escrow community and when I contacted them they said okay, we're sorry, we know there's a problem, we can't do anything about it right now, but as soon as our CDN is ready, we are.

The next group is a group of hosts basically used by the ruby communities, by all the GIT hosting providers, like GitHub and bit bucket, and Mozilla is also in that group. Contacting the projects resulted in the impression that they basically don't bother. I got a mail from GitHub telling me, we are aware of the issue but it has no priority for us at this time. The other projects, tickets sit in the queue for about months or even years without being accepted, commented, anything. Or being side stepped by answering questions that were not asked.

One common thing in this group is they are on items similar web services or the resale of items inweb services. It's a strange string Amazon web services apparently offered IPv6 in 2011 but has withdrawn that option in 2013 (Amazon) and it hasn't come back and I have seen no plans of it coming back. Okay.

So, the smaller hosts, at least four of them had enabled IPv6 by Friday. That's when I made these slides. In the meantime, it's another host hosting about 200 ports has enabled IPv6 on Monday. So, these numbers are slightly out of date.

And I think the other small group on the bottom will move to IPv6 in the coming months. So, the real question, the real road blocks are these two first groups. Before I take your questions, my question to you would be: How to proceed from here? Should we wait and hope that Amazon and firstly see the light eventually or should we lobby these Open Source projects to switch their providers?

Well, that's for me. For you, room for questions.

JEN LINKOVA: Questions?

AUDIENCE SPEAKER: Benedikt Stockebrand. The answer to your question is obviously yes. Basically we have to do all of this to make these people aware of things. Move our business away while we can, where it doesn't really hurt, and tell people why we do so roughly. And otherwise this is a night thing. Thank you.

AUDIENCE SPEAKER: We have a problem experiencing access in some of the repositories of free software and during the updating our systems, through the IPv6, and some of them are could providers, transit providers, it means that we probably have connected IPv6 backbone, this problem still exists but the main problem that tells us to sometimes even disable IPv6 for a moment, we update our systems, is that AAAA records seems fine but it's not working. The roots in the backbone come into black hole or something else, cannot find. We are started searching talking to NOGs and find out what they just don't care about it, and this is one of the biggest problems in our Open Source in IPv6 access. Please take care of your AAAA records and take care of your IPv6 abilities. If you are not available through IPv6, don't place your AAAA records. If you say A, say B.

JEN LINKOVA: Say AAAA you mean.

WOLFGANG ZENKER: Actually the view I gave you is maybe a bit overly optimistic, because we assume that everyone who has a AAAA record is reachable by IPv6. That might not be true in all cases.

JEN LINKOVA: Thank you. Now we got another interesting story for us, what happens about when you deploy IPv6 only network. We are slightly late, so feel free to leave the room whenever you want. We maybe slightly, five minutes late to the coffee break, or is it too late for coffee?

ROGER JORGENSEN: We are going to have a little talk about how to make trouble for yourself and that's what you do by making IPv6 only network. This Ola.

First we're going to zoom into where we are and that's in Troms, it's a bit further north than we are. 1,500 kilometres north. This is the size of the fibre network that we are building our IPv6 only network on top of. And we have failures regularly, shotgun hunting, this is just, it happens regularly.

And this is our helicopter that nearly took out of, well 50% of our cable, missing a power line. So, redundancy is important.

And Bredbandsfylket is the regional operator, we are not profit. And we are long term and stable ownership of a fibre opt I can network and we are providing network services to our owners. And we want to provide a long term and future approve network, 25, 30 years, counting from 2003, and we want a tool for our owners to provide efficient services and ‑‑ well infrastructure they can rely on.

And the first version that had IPv6 support lasted from 2006 until, well it's still running. So whatever you do now has to be future proof and cannot use private addresses because we know they are overlapping everywhere. We have between 1,500 ‑‑ well, you can read the text. And the important part here that is IPv6 related is that we are going to give some of our end users access to our, to the CPEs, so, we have to use something unique for them, and that's IPv6.

We had a public bit for this. A very long process, lots of discussions, and to get IPv6 into it was very, very hard. And it ended up with a contract in November with nLogic and we started to build this in January and we had it operational by April. We are going to put production ‑‑ well we had production on it, now we are going to put much more on it this weekend when I get home.

Now the fun part.

OLA THORESEN: I'm from nLogic and we got the task of building this network, while those guys have built the physical infrastructure, we were told to build the logical network on top of it.

We want to be a partner with our, both our vendors and our clients, so we build networks, we design networks, we support networks. And we help our clients actually what they want. And for us, IPv6 is really a core technology.

We believe that you don't have real Internet access if you don't have IPv6. That doesn't mean that we are able to do everything on IPv6. But we were very happy when Roger gave us a challenge, when he said that he wanted all management of all CPEs to be IPv6 only. We started with management of the CPEs, we're not going to go to where the last couple of speakers were, building an entire network based on IPv6, but at least management would be on IPv6. And actually when I got that task, I thought that that will be pretty easy. The equipment supports IPv6. So, of course that won't be a problem.

We built the entire network, we were very lucky, we were able to start on a greenfield. They had the fibres, the fibre cables but no equipment. So we built everything based on Juniper equipment which has good IPv6 support. MX in the core, AC X at edge and using MCI C X as the CPEs. They also support Juniper so they work on IPv6, that means that that shouldn't be a problem to make management on IPv6.

Unfortunately, we pretty quickly discovered that IPv6 ready does not equal IPv6 only ready. There are lots of bumps in the road when you try to build an IPv6 network. As we heard earlier today. But we wanted to do it using standard technology and automate all the processes to avoid human errors, because we know that human errors are maybe the most common way of making networks fail.

So, we wanted to deploy what Juniper calls and what everyone calls zero touch provisions which basically means that the CPE boots up, gets its IP address from the D PCP, gets the values to download the config, down leads the config, everything is up and running, nice and dandy. Except that there is of course no support for IPv6 or DHCPv6 in that process. So that means that we need to build at least a temporary IPv4 network to actually deploy the software or deploy the config on the CPE in the first place.

So the next problem we discovered was that, all right, the CPE supports a static IPv6 address on the management interface, it supports slack, but it does not support DHCPv6 addresses on the management interface, they know to know which CPE is on which location, so they need control over the addresses so it gave us no option other than using static IPv6 addresses, which means all CPEs needs unique configs, that was bump number 2. We needed a way to fix that.

So the solution to say two two problems was that we installed the new server, it's pretty nice. Pretty better. But it works for this as it only deploys the config for the CPEs. I actually had to write a hook or a plug infor the server to make it support the options we needed. But it's pretty nice. And it means that the server creates a config for the, or the DHCP creates a config for the DHCP and in that config, IPv4 is disabled from the CPE management and the CPE is available for IPv6 only.

So then we had the next problem. To manage all the CPEs, we installed the Junos space management platform. It's a decent management platform. Not the best out there but it's pretty good. It has full support for IPv6 in principle. The problem is the way it discovers new devices. That's obviously made for IPv4. You enter a sub‑net into the config and say that scan this sub‑net for new device devices regularly, which of course works pretty nice in IPv4 world. Does not really work well in IPv6. And there is nobility infrom Juniper to allow the device to self register as it boots up or comes online into a Junos space solution. So we need to fix that.

Fortunately the platform has a nice API, and the hosts, the Junos host has a nice scripting language. Which means that using that scripting language, we can actually send out a discover me please API request, or discover this IPv6 address please using Curl from the actual CPE as soon as it comes up. So that's nice. But how do the CPE know that it's actually reachable?

Well Junos has thing called event scripts which triggers a script as soon as something happens, an event happens. It can trigger on ping, only IPv4. It can trigger on an HTTP request which can try and reach out to the Junos space server, but that only works over IPv4, and we needed to wait until the host was up and IPv6 only. So that doesn't work. We can't actually understand that it's received rip NG routes or anything else so we ended up using a simple timer. It's fragile but it works, as soon as the box comes up with IPv6 only, every 60 seconds it will try to connect to the Junos space platform. It does that for five minutes and then gives up. Answered this you can always manually do that later if you need to.

But, it works. So, again we found a bug, or we found a problem and we worked around it. Which is basically what you need to do whenever you try to build and IPv6 only network.

Next problem was radius. They use radius for authentication when locking into all the devices. We found two bugs there. We run free radius on a Ubuntu 14.04. There is an old kernel bug, which means that the kernel drops IPv6 radius packets, or actually drops large UDP packets or something, pretty strange. Also, as we can see from the radius configuration on the actual switch, pretty common error, that's been fixed fortunately in the newer versions but in old versions they used a colon to separate both parts of the IP address and the port. And also, they had for some ransom IPv4 address as the last part of the operatedious configuration. So stupid bug, it means that radius doesn't work until the CPE is upgrade to a newer version, again that's just a matter of a few clicks, upgrade to a new version and you are up and running and radius is working.

So now I have a situation where we can actually deploy a CPE. Because of the few issues I mentioned here it meant I have a little bit of work to actually do it. It's not a hundred percent zero touch. They have to create the config, which means that they have to deploy a new CPE. They had to add it to DNS and they get some cut and paste config ready to paste on the the PE routers. We could add that to the PE routers when that comes but we're a bit scared to do that so currently they are using cut and paste for that but it might be added later.

But the end result is that you can now actually deploy a new CPE, the in five simple steps. Registered in the previous February, unboxed the box, installed it, connect the cables, boot, wait for a few minutes and the CPE is ready. It is already configured. It is remote manageable only over IPv6. SSH works, net could have, everything works, all the (net cough) ready to use forth customers, the this, PE is available, they can log it and see that everything is working. And all they need to do is upgrade it to the latest software so get radius authentication to work as well.

So in the end I think that we managed to find quite a few problems but we also managed to work around them. And I think that the most important thing here is, what we learned in the process. As has been said before, and that it is actually possible, and this is a production network and the management here is of course pretty important for them, but of course they don't have to think about all the home users or anything, because it's just a management network. But, I think that if you want to do something like this, you should just try to do it. You should just test it in a lab of course if you have to, but just try it, because it is possible, and to answer one of the questions that came up earlier, why would you want to do that? Why would I want to break my network running IPv6 only? We only have to run IP stack in that network. We don't have to have access for both IPv4 and IPv6. We don't have to manage firewall rules, routes, we only have one IP stack and you should choose one IP stack. I think you should try to choose IPv6. Use IPv6 whenever you can and use IPv4 if you must.

So, that's what I want to tell you

JEN LINKOVA: Questions?

AUDIENCE SPEAKER: Hi. I wish to thank you for your great work and this was very bold step I would say. And in terms of size of the implementation, I know that we with all due respects to luke's work and things that he did, we are hearing about home implementation since 2012, and that is what I would say, same opinion of Randy's tool, so there's an elephant in the room and we're not talking about it, and there are big steps that need to be done. I'm guessing you are all waiting for Apple to release new iPhone IPv6 only, that way everything will work tomorrow morning.

OLA THORESEN: EIX phone got upgraded just now, at least in Norway all i‑phones are on IPv6. But I know what you mean.

AUDIENCE SPEAKER: Thank you. Sorry, a question for you and RIPE NCC also. And I do apologise if I missed that, but I didn't see a new adoption numbers, I am kind of used to adoption numbers and rates for IPv6 technology, so did I miss that or there is a graph that we will see?

JEN LINKOVA: Okay. I got that. I probably need to put that option graph on every presentation I do, right. Probably should do it as an opening slide for the session. Last time I checked it was 10% by Google measurements. I do not know like statistics for local IPv6 only network, we might ask our RIPE people to give us something, but sorry, I didn't prepare any slides about IPv6 only network here at this time. I assumed it just works.

GERT DÖRING: It just worked.

AUDIENCE SPEAKER: Tom hill. Thank you very much again just to echo what everyone else has said for taking bold steps to do this. About two or three years ago I actually tried to do something very similar in our new data centre and I gave a very similar presentation at UKNOF 29 and came to very much the same conclusion that IPv6 only ‑‑ IPv6 ready does not mean IPv6 only ready. And I found a litany of problems with different vendors, AP CPUs extreme network switches, all sorts of things. The one thing I would say to this is wherever you found these problems, please please please feed them back to your vendor, it does take time and effort, I realise that but I have had some success reporting these problems back to them. It's not perfect and we don't have a massive spend so the more people that do this brilliant, if you can put it in a tend Dear document it must do IPv6 only and you are spending a few million that would be fantastic. The more of us that get into this and hammer into their head that they need to support this, not in a sort of 50:50 manner, that would be fantastic, but thank you for your feedback it was very interesting.

ROGER JORGENSEN: I must apologise to the vendor because I tricked them into listening to this talk without telling them this much about it, hopefully they will fix some of these basic things.

AUDIENCE SPEAKER(Gert Doring): I actually came here to comment, and I appreciate that you brought the feedback. And that, the most important message I hear from your talk is, there are still road blocks, but it's possible to do it. So, sometimes you have to be a bit creative and sometimes it's totally annoying and needlessly frustrating, but in the end you can do IPv6 only today. So there's no more excuse to stick to v4 only. Thank you.

ROGER JORGENSEN: We didn't really have a choice really, because I think there is ‑‑

GERT DÖRING: You could have done what everybody else do like 100.64.

WOUT DE NATRIS:

WOUT DE NATRIS: There's a duck sitting on your error. This is really, really good. Appreciated.

AUDIENCE SPEAKER: I am Chris talking for myself ‑‑

ROGER JORGENSEN: I wanted to reply to Gert. We didn't have a choice actually because this is Version 2, it has to live for more than ten years, there was no option really.

AUDIENCE SPEAKER: I am Chris talking Tor myself. I need to set up some v6 test environment using older Juniper equipment, so if the vendors are listening, everything was version 12, version 13, version 14 breaks in many many things and if you don't have equipment you can't upgrade to version 15 we're just sitting on a bunch of crap. Okay. That's my synopsis.

AUDIENCE SPEAKER: Hi. Natalie Trenaman, RIPE NCC. I just wanted to come back to the comment that we had earlier why there was no update this time from me. And that is because I saw the agenda and I was duly impressed with the stories that we were, that were on the agenda already, because I do prefer to hear real life operational stories over statistics, and but I do see that people would like to see statistics, so I'll be back next time. By the way, if you don't have the five star hype T‑shirt yet, I brought some.

JEN LINKOVA: Great. Thank you. Any more questions? May I ask two. So am I understanding everything else like syslog, NetFlow, whatever the protocols, management protocols you you are able to enable in the blocks.

OLA THORESEN: We only run the CPEs on IPv6 only, so, there's no NetFlow going from them actually. But everything else is working fine. I think NetFlow should be working fine as well. I think actually Juniper at least with version 15, most stuff is ‑‑ most of the basic stuff is working and the second set of basic stuff. But there are some things they haven't really thought about and some things are a bit more difficult to configure and especially that you can't actually DHCP on the management interface and some stuff like that, but all the protocols are working as far as I know. So, yeah... we would like to, and we can't for various reasons which I can discuss off line afterwards, we would like to also run the core network management on IPv6 only, but that has to do with how I built and designed the network so we can't actually do that. But that's one future upgrade we want to do maybe later.

JEN LINKOVA: And my second question, so you did not try to DHCP, you just wanted one single address to send a mail right? I mean, just get an address for device not delegating a prefix to the whole box. Because I suspect that might actually work.

OLA THORESEN: It won't get an address. It just doesn't work.

JEN LINKOVA: Thank you. And we are on time. Thank you very much. Please rate the talks. And please go to the BoF which is in the tutorial room. I was asked to remind you, and thank you very much for your attention, please send us your suggestions, submit your talks for the next meeting, and see you in Madrid.