After 39 days, 7,863 miles, and scores of contacts with WISP’s, owner/operators and users of MikroTik, Ubiquiti, and other disruptively priced, high feature networking gear, the wireless roadshow will take the checkered flag July 13-14 in Dallas, Texas by staging a two day MikroTik MTCRE training.  A couple of seats remain in this class and you can sign up HERE.

It has been a wild ride, but I have seen so many diverse applications for the equipment we have built our business on, used in creative ways to enhance the lives of those we serve.  From RV Parks supplying WIFi to travelers, to boats that provided WiFi in a very mobile way (on the water, definitely no wires here) to impossible point-to-point links through trees I would have never tried, we took in sites and experiences that I will never forget.  Along the way we met some of the kindest Americans and Canadians, who welcomed us into their towns.  Thank you all.

I will have a few more posts between now and Dallas next week as I take a few days to prepare for class and catch up on some paper work, and will wrap up the trip after the Dallas training.  Thanks for following our journey, and for being our customers.

Today in St. Paul, Minnesota (yes, we made it there yesterday) working on a customer’s Ubiquiti NVR I was faced with one that was continuously recording from 20 cameras.  Needless to say the storage was not lasting.  I added an external USB drive using my standard process.

UniFi-Video – How to add an External Hard Drive to the UVC-NVR or airVision-C (Hardware NVR)

All done, time for a little relaxation!

Today I took a support call from a customer who was having strange issues with their TDS Cable modem service.  Http pages would load fine but https pages had broken images, broken style sheets or would not load at all.  Itunes was flaky, apps using https were flaky.  It seemed to be many different problems and then I remember the first time I implemented PPPoE in my first WISP, Wickson WIreless.  Same issue and it was MSS or Maximum Segment Size and MTU.  I will leave the discussion of PPPoE out of this post but here is how the issue occurs.

A post from Stretch at packetlife.net does a reasonable job of explaining the issue, I quote: ”

When a host needs to transmit data out an interface, it references the interface’s Maximum Transmission Unit (MTU) to determine how much data it can put into each packet. Ethernet interfaces, for example, have a default MTU of 1500 bytes, not including the Ethernet header or trailer. This means a host needing to send a TCP data stream would typically use the first 20 of these 1500 bytes for the IP header, the next 20 for the TCP header, and as much of the remaining 1460 bytes as necessary for the data payload. Encapsulating data in maximum-size packets like this allows for the least possible consumption of bandwidth by protocol overhead.

Unfortunately, not all links which compose the Internet have the same MTU. The MTU offered by a link may vary depending on the physical media type or configured encapsulation (such as GRE tunneling or IPsec encryption). When a router decides to forward an IPv4 packet out an interface, but determines that the packet size exceeds the interface’s MTU, the router must fragment the packet to transmit it as two (or more) individual pieces, each within the link MTU. Fragmentation is expensive both in router resources and in bandwidth utilization; new headers must be generated and attached to each fragment.”

Fortunately, the internet has a remedy for this problem in RFC 1911 and it is called Path MTU Discovery or PMTU.  The RFC explains the process in detail but let it suffice to say it is not perfect because of the way hosts on the internet behave.  Stateless load balancers for example are unable or unwilling to respond properly to the ICMP messages generated by Path MTU Discovery, thereby breaking the mechanism.  Guess where this technology and effect is seen most often?  You guessed it,  popular sites like  banking, secure sites, community.ubnt.com for example, basically any site that pulls URL’s from other sites, runs SSL or uses load balancing technology.  I am sure there are many more examples but you get the idea; if images are broken, random https sites won’t load and apps behave strangely or erratic, it is likely broken Path MTU Discovery.

Now, the most important thing, how do we fix?  I started by calling the provider.  I was escalated to an engineer who first blamed my router and modem (surprise) and then after much clicking of keys (maybe he was playing a game) he said he didn’t know how to fix it or if there even was a problem.  Thank goodness there was  MikroTik router on the client end of the link.  MikroTik (and Linux) have a feature called “Clamp to pmtu” in mangle.  This feature dynamically changes the MTU settings to match the smallest MTU from point to point and thereby prevents fragmentation and he weirdness I previously described.

Here is my fix:

 

In Winbox, click IP Firewall Mangle and create a new rule for packets leaving the WAN interface as follows:

On the Advanced tab set the matcher to match Syn packets:

On the action tab, set the action as follows:

Repeat for a second rule to match packets coming in the WAN interface as follows:

The end result is this:

If you follow this step by step and insert your WAN interface for ether1, you will fix the problem and be a hero.  Good luck diagnosing and solving PMTU problems!

 

First, I have found all Canadians to be warm and inviting, fund and very welcoming to us.  There are a few things that I have observed that I thought might be interesting to share about French Canadians (who are particularly fun to hand around with) in particular:

1. They like neat, mowed and clean yards and will go to great lengths to make them that way.  East Texans, a lesson learned!)
2. They like red, and many things are painted red including their roofs.  it is is not red, it is some other bright color.
3. Many speak only French which is cool.  It is interesting to me that they have preserved their language when surrounded by English speaking provinces and countries.
4. All houses, even the tiniest are neat and perfectly landscaped.
5. They prefer small, neighborhood markets over large super markets.
6. Most have their own gardens and there are many roadside stands to buy fresh vegetables (les légume).
7. Many enjoy an afternoon glass of wine outdoors in a stemmed glass (because the climate here is so mild, outdoors is the place to be).  We decided to participate in that custom!
8. When you pass over the Quebec border, instantly everything goes French.  The signs, the markings on the packages at the store, the business names, and it seems even the markings on the trucks passing me  on the highway mysteriously change to French.
9. They gather apples frozen hard on the ground in fall/winter on Île d’Orléans, Québec and convert them to cider.  Cider here isn’t your grandmother’s cider, it’s a sweet wine, almost like a brandy.  The freezing of the apples boosts the potency of the juice and the alcohol content to about 12%.  So, as the lady at the cider boutique told me “be careful with this one, it is dangerous” pointing to the bottle of strawberry flavored cider in my basket. She was correct.  Nuff said.
10. I will not miss the bagged milk, give me a non-earth-friendly plastic gallon jug any day!

Tomorrow we leave Canada and the road show turns south through the northern USA.  It will be good to be home where gas costs about half what Canadian gas costs, and the prices are more like what I am used to but I will miss our Canadian friends we met along the route, the great food, fresh vegetables, hearing “bonjour” and “merci boucoup”in the sing song way that makes it sound so cool, and all the other things I have learned to love about this place.  I will be back!

To my customers I met with along the route, thank you each for your hospitality and for being our loyal customers.  Your business is appreciated and never taken for granted.  I will see you all soon!

Wisconsin, here I come!