Setting up FreeBSD and jails on Azure – part 1: networking

I set up this blog on Azure as an excuse to play with the new FreeBSD VM Depot image, learn more about jails and write the occasional blog post about random stuff. I took extensive notes while at it and I will be posting them here for future reference and to help the occasional search engine user.

I will skip all the clicking through that can easily get to a running FreeBSD VM in Azure. There is tons of FreeBSD documentation, including specific Azure tutorials that my team and others have written. I am lazy, so I will just point out specific Azure differences and how to take care of them.

A word of caution: please don’t consider what you read here to be authoritative. I’m doing this for fun and my free time is what it is, so don’t think I researched this stuff thoroughly. It worked for me and seems to be still working as I write this – that’s all I needed.

Let’s start with networking. Every public cloud has their own approach, and Azure is no different. Two things to remember about Azure IP management:

  1. Your VM will have an internal IP that will be translated to a public one. If you are puzzled about the number not being in your expected 192.168.0.0/172.16.0.0/10.0.0.0 ranges, know that you are looking at the wrong RFC: the IP will come from a “Carrier-Grade NAT” pool specified in RFC6598. And, unless you pay more set up a VNET, it will change between reboots. Being the cheapskate I am, I didn’t bother. But here be dragons, so stay tuned.
  2. Your external IP is again assigned dynamically, and dyndns’d to [yourmachine].cloudapp.net. Again, for an additional (and actually minimal) charge you can reserve a block of IPs, but given that the IP will be reserved for the lifetime of your deployment (that is, it will survive a reboot), you may well not need to pay a premium. YMMV.

With that in mind, I quickly stumbled into an issue. Jails on FreeBSD require all the services in the host machine to be bound to specific interfaces. That is you cannot bind to 0.0.0.0, *.*.*.* or any general address. Ignore this, you may end up like me – locked out of your machine as soon as you start the first jail (damn you syslogd!).

‘sockstat -4’ is your friend here as it will list which services are bound to which interfaces. On my vanilla install I had to change syslogd and sshd as they both are configured to listen to everything and the kitchen sink.

Syslogd was a piece of cake as long as I was fine with loopback only – it may become trickier if I need to get fancy and gather logs from the network. Add this in /etc/rc.conf

syslogd_flags="-s -b 127.0.0.1"

restart syslogd and you’re off to the races. Do not try ‘-s –b [yourhostname.cloudapp.net]’: intuitive as it might seem, the DNS lookup will give you the public IP address for your host, so you will be out of luck.

SSH was more of a pickle. The ListenAddress directive in sshd.conf will accept either an explicit IP address or 0.0.0.0 to listen to every IP on your machine: there is no way use an interface name such as hn0. You can specify a hostname, but then again a DNS lookup for hostname.cloudapp.net will give you the useless public IP address so you would be out of luck.

The cleanest hack I could think of was using /etc/dhclient-exit-hooks to update sshd_config with a new ListenAddress and restart sshd with every IP changes. The first version of the script looked quite brittle for such a delicate task, so I thought I’d leave it for a rainy day and moved on to something simpler based on only two moving parts:

  1. The internal IP will not change as long as the machine is up, so it’s reasonable to expect that adding a ListenAddress directive using the current IP will be quite stable. An invalid IP address will not prevent sshd from starting so I can live with it to take care of my port 22 to begin with.
  2. Belts and suspenders come handy when it comes to logging on your machine , so it occurred to me that I could use PF (yes, I like it better than ipfw) to bind SSH to the loopback address and redirecting incoming traffic from the public interface over there. This way, if the IP address changes, I can still connect using a different port (in the example below it’s 2222).

Simply enough:

add a ListenAddress line to /etc/sshd.conf:
ListenAddress 127.0.0.1

enable pf in /etc/rc.conf:

pf_enable=”YES”

add a redirection rule on /etc/pf.conf
#INTERFACES
ext_if="hn0"
rdr pass on $ext_if inet proto tcp to port 2222 -> 127.0.0.1 port ssh

start the service
service pf start

Restart sshd, cross your fingers, open a new terminal and try to ssh in again. Do NOT, for the love of what’s dearest to you, close the shell you’re in as it could be your only way back if you did something wrong. After testing that this setup works, I have closed the endpoint on the Azure firewall, but reopening it will be a matter of seconds in case things go south.

(For the record, things went south and I got locked out nevertheless. Twice. Stay tuned as I will make copious fun of myself in an upcoming post, together with providing a few survival tips to recover a deployment gone belly-up.)

The final somewhat annoying bit of having a somewhat ephemeral public IP address is that you need to be careful with naked domains. I’m not a fan of naked domains per se, but at the same time I understand there is a general expectation that http://boldlyopen.com works just as well as http://www.boldlyopen.com so I will be going with the flow.

Short of the DNS powers that be finally realizing that it’s 2015 and the limitation on CNAMEs is something they should work on, the only solution is to bite the bullet and use the public IP Azure is giving you. I could have bought a few IP addresses for a very reasonable amount well within my MSDN subscription, butI care about IPv4 space so I didn’t feel my blog should squat one of the few left. Then again, while this IP may change, it is guaranteed to remain the same throughout the lifetime of the deployment and in my case I have no plans of flip-flopping this machine at all. I made a mental note to update the DNS in case I ever start from scratch and I got on with my life.

With these quirks out of the way I was finally able to wade my way into the world of FreeBSD jails: a topic for a follow-up post.

Hello from FreeBSD and Azure

This blog is now running on a set of FreeBSD jails on Azure. Setting it up was relatively easy and smooth, modulo a few bumps on the road that I have documented and will share in upcoming posts.

Why FreeBSD and why jails, you may ask? Everyone and their dog seems to be running Docker containers on Linux these days, so going FreeBSD can legitimately raise a few eyebrows. I wish I had a compelling answer and I could start pontificating about performance, security, scalability and the like but this is not the case: my blog doesn’t get nearly enough traffic to even start discussing scalability and I’m not nearly enough conversant on system internals to provide guidance one way or the other.

Truth is, I just resonate with FreeBSD more than I do with Linux. I cut my UNIX teeth on Linux, but I quickly moved to FreeBSD after my Linux workstation was pwned some 15 years ago. I came to FreeBSD for PF and stayed for make world.

Back when Linux was busy creating amazing things and paying a price in terms of slight yet annoying incompatibilities and quirks, FreeBSD has always been a cornerstone of predictable, solid performance. And usability too: hier is a joy, and knowing that anything that is not part of core goes in /usr/local and that port maintainers will leave configuration files alone (Debian, I’m looking at you and your a-little-too-clever Apache’s sites-available) takes a lot of guesswork away.

Over the years I deployed on Linux a number of times, mostly because I needed JVMs and as there wasn’t just enough choice of FreeBSD VMs out there. I did however make a point of being a pain in the backside of many colleagues at Microsoft until we got FreeBSD on VM Depot. The cherry on the cake was some downtime over the recent holidays which allowed me to learn my way around jails and successfully migrate my aging Linux machine over.

You are now connecting to FreeBSD 10.1, with four different jails running a DNS server, a MariaDB service and a couple of websites running mostly WordPress. A testament to FreeBSD quality is that to set it up I just had to dust some memory shelves and learn a couple of new things: despite not having deployed anything serious in the best part of 10 years, things are pretty much the same as they used to be, and the utmost predictability of what’s going to happen on a FreeBSD system is still very much there.

Please don’t get me wrong: it’s not like I don’t like Linux – I obviously did and still do although with many reservations on systemd. It’s just that one way or the other I keep coming back to FreeBSD: home sweet home.

So where have I been?

TL;DR – here:

That was a lot of miles!

Have open source, will travel. And travel more than what the map shows, as I have been multiple times in many of those countries.

Thing is, I haven’t stayed long. With the exception of China, where I usually stay for a whole week, I try to jam-pack as much as I can in my travels, and that usually means I’m in a country for an average of 36 hours before I move on. The craziest ones are the “three strikes” days: wake up early in country #1, fly to country #2, fly out in the late evening and sleepcrash in country #3. My back still holds a grudge for doing Sweden, Finland and Portugal this way.

As counterintuitive as it may seem, this travel pattern while positively crazy is incredibly productive. While I started to travel this way so that I could minimize time away from my family, I quickly understood it helps immensely to focus and get things done effectively. It does take a toll on planning, but there is a whole lot of things you can do in one day: my daily fare typically includes at least a few internal meetings, a couple of customer/partner visits, the occasional press roundtable and a talk at a community event.

It is frantic indeed, and sightseeing is pretty much out of the question unless it’s a drive-by on the road to or from a meeting. I do however spend lots of time with local colleagues, and that alone helps a lot understanding the place I’m visiting: many tourists may spend weeks in the same place, visit every corner, and yet fail to understand what a country is all about as they lack contact with local people. Me, I will forever cherish the two hours I spent last Monday with 40 students from Egypt, learning so many things about their country.

Another great way to learn about a foreign place is food. It takes some convincing, especially in far away countries, to talk colleagues into ditching the typical 5-star hotel international buffet for a local dive place, but it invariably pays off. I have very fond memories of a Padang place in Indonesia, and I will positively refuse to leave Singapore without a pilgrimage to the Maxwell Road hawker stalls. During this very last trip I discovered Ful Medames and Mulukhiiyah, and I am invariably looking forward to what China will have to offer in my next trip. For the germophobics reading this, know that in four years I have only been sick once, and that was when I couldn’t bail from a lunch at an Italian place in Beijing.

Then again, being with local people makes all the difference. And having a job where I get to meet so many local communities takes all the wear of travel away. I am sitting in my last hotel room in South Africa as I write this: I landed at 5.30am, gave an opening speech at an open source event at 9am, rushed to the office for a PR roundtable, shot a video and had a few internal sync meetings. As I try to pick myself up for my last day on the road, and then finally go back to my family I can’t help but thinking how lucky I am.

A tale of two countries

It boils down to shoes.

A lot of people asked me what it feels like to be an Italian in the USA. Books have been written, pictures have been taken and videos have been shot, so I feel I should spare you the details on getting Italian ingredients (you can get everything here), wrapping our heads about pounds and gallons (seriously, what were you guys thinking?) or remembering to have a full tank before hitting the outdoors (and a bear-proof food container if you plan to camp). Also, we are talking about a country that spans several time zones, so if there is no point trying to lump things up.

I do however have a story for you and I believe it pretty much sums up our experience in the Pacific Northwest, the corner of the States we have been lucky to dwell for the past four years.

A few months before leaving Italy, I needed new shoes. Nothing fancy, just an easy pair of loafers for the summer. I went to a shopping mall nearby, found what I wanted in a mid-level shoe store, paid up and off I was on my merry way. After a couple of days my new shoes split wide open, a tragic structural failure that had me limping home and later to the store to voice my disappointment.

They asked to see the receipt. They wondered if I did something weird with the shoes (though I don’t exactly look like I’m into parkour), they had me request to speak with the manager and finally they offered to repair, not reimburse – not replace, the offending shoe. They took their good time and after a few weeks I picked up a hack job of hot glue and amateur stitching that of course found its way to the trash bin shortly thereafter.

Fast forward a year or so, when I found another pair of loafers in a middle-of-the-road outlet store in North Bend. I liked them, tried them on, bought them and off I went. In a few weeks a small elastic band ripped off: it didn’t impair functionality, it was mostly cosmetic and a minor issue altogether. I didn’t bother going back to the store and actually kept that particular pair as my go-to shoes for quite a while. As a matter of fact, I still had those shoes on when I was back at the very same store almost a year later.

When the store clerk approached me, I told him I really liked the shoes I bought a year before, and I was wondering if they had something similar. He took a look at my feet and immediately asked what happened as he noticed the small rip in the fabric. I told him it was no biggie at all, but he insisted on seeing the shoe, asked when I bought it and told me in no uncertain terms that he was going to replace it under warranty. I told him I didn’t have the receipt, and he just asked for the credit card I used for the original transaction.

Minutes later I was driving home in my new free shoes, both me and my wife stunned at what just happened and commenting how good business sense, trust and, more generally, a positive attitude can make all the difference in the world. We love this place.

And so it goes…

You stop updating your blog. A few weeks go by, then months, then all of a sudden you realize it’s been years since you wrote anything. Starting again is hard, as you feel you should write something important, explain why you have been away, condense years of stories into a big catching up post, commit to a New Year’s resolution of getting back to blogging and try to stick to it – the works.

Or, you can just rediscover how much you really like FreeBSD, play with jails for a while, spend some time migrating a couple of databases and thinking that after all it’s just a blog, the world keeps on rolling with or without it and what is four years without posting among friends?

So here is to a fresh install of WordPress on the latest FreeBSD, all running on Azure (more in upcoming posts). And to whatever may come next. I’m not promising it will last, but it feels good to be back.

Off to Europe: coming to a city near you, and looking for beers!

As mentioned in my previous post, I am off to Europe this coming Saturday, visiting four countries in six days (yikes!). More specifically I will be:

  • In Milano, Italy, Sunday 30th, Monday 31st and Tuesday 1st. Would love to hang out for beers and some great pizza (have been missing that) on Monday. Anyone?
  • in Munich, Germany, arriving Tuesday 1st and leaving the day after. I’m on an early flight and I count to hit downtown for food and beers – make yourself heard!
  • in London, UK. arriving on Wednesday 2nd, leaving on Friday 4th. Planning a serious pub crawl on Thursday, care to join?
  • and finally in Brussels , Belgium on Friday and Saturday. I plan to attend FOSDEM of course, but I’m also considering checking out how the Brussels venue of Amadeus compares to their Ghent counterpart. Anyone up for a real man serving of spare ribs?

It will be quite a trip, packed full with meetings and a lot of plane hopping. Looking forward to coming back and meet old friends on the other side of the pond!

Three months in the whirlwind

Well, hello world! It’s been quite a while since my last post, and every single day since then has been a crazy, exciting and furious whirlwind. Since October 18th, my life has al been about relocating the family to the US and getting acquainted with my job in Microsoft. And that makes two crazy tasks rolled into one: relocation has been overall smooth – and the Microsoft relocation team are nothing but angels – but still it has been a humongous list of big stuff and gory details, from preschool arrangements to getting SSNs and driving licenses, from buying a car to getting used to “italian sausage” with fennel seeds. Luckily enough we just found a house and our goods should be on a ship by now, so if everything goes as planned it will be housewarming party time sometimes in early March. That will be a huge load off my back.

Work-wise it has been even crazier. I cannot even begin to describe how this company is huge, multi-faceted and incredible in many ways. For this Italian guy coming from small and medium companies, adjusting to the different pace and level of interaction has been a very interesting journey. It is still striking after three months to see how Open Source is all over the place. When I first joined, I was given the task to meet the people in the company with a stake in Open Source: three months have come and gone, my agenda has been beyond ridiculous, and I’m not done yet. By far. It seems that every single time I meet someone, I am given at least two more names of people I should talk to. I am now about to pack for a trip to Europe (will write a separate post shortly) and I know that will mean a few more notebooks packed with information, and another pile on the todo list. The good news is that all of that will be interesting stuff.

I sure hope to see a lot of you in person as I travel the world in the upcoming months. In the meantime, if you fancy a nugget of what I am here about, here is a short video of a roundtable that has just been posted on Channel 9:

A sad day for Open Standards

First things first: I don’t care about Apache Harmony. It’s not part of my daily life, it doesn’t pay my bills, I never downloaded it, I’m not a member of the community and I don’t even have an Android phone. In fairness, I couldn’t care less if the project is abandoned. Even more, I will readily admit there is a positive side in IBM ditching Harmony and joining OpenJDK, as the world is now closer to enjoy a strong Java platform.

The problem is the price tag. With IBM surrendering to the Oracle bully, the Java Community Process is now as credible as Weekly World News, and basically nobody is safe. The spin pros have been busy focusing on a strengthened, renewed Java effort, and they conveniently (or should I say pragmatically?) forgot to mention how dangerous it is to be under the illusion that the JCP is a neutral and cooperative body producing Open Source friendly specs when the truth is Oracle can and do whatever they want, including breaching the JSPA and getting away with it. Or play puppet master even with mighty IBM. I wish all my FSF friends will soon recover from the initial excitement for a GPLed Java and realize how, really, the party is over and we have much less freedom than before. And maybe a better JVM with no competitors – but is it worth the price?

There is nothing worse than living an illusion: if you still believe Java is free just because there is a GPL JVM out there, a rude awakening lies ahead.

Redmond, here we come

This will be a very short post, as life is incredibly hectic here. Let me just confirm that indeed I am joining Microsoft, and I am overly excited at the idea.

On top of an incredible amount of warm wishes and kind words, I have been asked a number of questions I really want to answer, and generally speaking I really should find some time and dump my thoughts. Just bear with me for a little more, while I frantically try to wrap up 41 years of Italian life and get ready to move across the pond. You will hear a lot from me, promise!

Time to move on

Now that the important news is out (and starts sleeping at night), the time has come for another announcement: as of May 4th I have left Sourcesense.

To many this might come as a shock, and I have troubles believing it myself: after all, I founded the company five years ago, so my departure has special meaning and lots of emotional baggage. It was not an easy decision by any means, yet a number of factors made me realize the time had come to move on and start looking for my next endeavor. And, in the meantime, enjoy my newly enlarged family: I am now a quasi-full-time dad, looking forward to spending the summer with my beloved ones, and I have never been happier. Sourcesense remains a cool company to work with and for: it is now in the capable hands of Marco Abis and you should definitely get in touch if you are looking for kick-ass consultants or a great place to work.

What’s next for me, then? Truth is, I don’t know and, in fairness, I haven’t been hunting for jobs in any serious fashion thus far: I do have a few very interesting talks, and I’m expecting some of them to turn into actual offers, and I am also toying with a few ideas to start something afresh (gotta love that new startup smell). But I do intend to take my time, and some rest as well. And yes, I will keep this blog posted with my progress: in the meantime, please update your addressbooks and know that g.rabellino@sourcesense.com isn’t going to last, so if you need to get in touch the safest bet will be gianugo@rabellino.it. More to come!