Setting up FreeBSD and jails on Azure – part 1: networking

I set up this blog on Azure as an excuse to play with the new FreeBSD VM Depot image, learn more about jails and write the occasional blog post about random stuff. I took extensive notes while at it and I will be posting them here for future reference and to help the occasional search engine user.

I will skip all the clicking through that can easily get to a running FreeBSD VM in Azure. There is tons of FreeBSD documentation, including specific Azure tutorials that my team and others have written. I am lazy, so I will just point out specific Azure differences and how to take care of them.

A word of caution: please don’t consider what you read here to be authoritative. I’m doing this for fun and my free time is what it is, so don’t think I researched this stuff thoroughly. It worked for me and seems to be still working as I write this – that’s all I needed.

Let’s start with networking. Every public cloud has their own approach, and Azure is no different. Two things to remember about Azure IP management:

  1. Your VM will have an internal IP that will be translated to a public one. If you are puzzled about the number not being in your expected ranges, know that you are looking at the wrong RFC: the IP will come from a “Carrier-Grade NAT” pool specified in RFC6598. And, unless you pay more set up a VNET, it will change between reboots. Being the cheapskate I am, I didn’t bother. But here be dragons, so stay tuned.
  2. Your external IP is again assigned dynamically, and dyndns’d to [yourmachine] Again, for an additional (and actually minimal) charge you can reserve a block of IPs, but given that the IP will be reserved for the lifetime of your deployment (that is, it will survive a reboot), you may well not need to pay a premium. YMMV.

With that in mind, I quickly stumbled into an issue. Jails on FreeBSD require all the services in the host machine to be bound to specific interfaces. That is you cannot bind to, *.*.*.* or any general address. Ignore this, you may end up like me – locked out of your machine as soon as you start the first jail (damn you syslogd!).

‘sockstat -4’ is your friend here as it will list which services are bound to which interfaces. On my vanilla install I had to change syslogd and sshd as they both are configured to listen to everything and the kitchen sink.

Syslogd was a piece of cake as long as I was fine with loopback only – it may become trickier if I need to get fancy and gather logs from the network. Add this in /etc/rc.conf

syslogd_flags="-s -b"

restart syslogd and you’re off to the races. Do not try ‘-s –b []’: intuitive as it might seem, the DNS lookup will give you the public IP address for your host, so you will be out of luck.

SSH was more of a pickle. The ListenAddress directive in sshd.conf will accept either an explicit IP address or to listen to every IP on your machine: there is no way use an interface name such as hn0. You can specify a hostname, but then again a DNS lookup for will give you the useless public IP address so you would be out of luck.

The cleanest hack I could think of was using /etc/dhclient-exit-hooks to update sshd_config with a new ListenAddress and restart sshd with every IP changes. The first version of the script looked quite brittle for such a delicate task, so I thought I’d leave it for a rainy day and moved on to something simpler based on only two moving parts:

  1. The internal IP will not change as long as the machine is up, so it’s reasonable to expect that adding a ListenAddress directive using the current IP will be quite stable. An invalid IP address will not prevent sshd from starting so I can live with it to take care of my port 22 to begin with.
  2. Belts and suspenders come handy when it comes to logging on your machine , so it occurred to me that I could use PF (yes, I like it better than ipfw) to bind SSH to the loopback address and redirecting incoming traffic from the public interface over there. This way, if the IP address changes, I can still connect using a different port (in the example below it’s 2222).

Simply enough:

add a ListenAddress line to /etc/sshd.conf:

enable pf in /etc/rc.conf:


add a redirection rule on /etc/pf.conf
rdr pass on $ext_if inet proto tcp to port 2222 -> port ssh

start the service
service pf start

Restart sshd, cross your fingers, open a new terminal and try to ssh in again. Do NOT, for the love of what’s dearest to you, close the shell you’re in as it could be your only way back if you did something wrong. After testing that this setup works, I have closed the endpoint on the Azure firewall, but reopening it will be a matter of seconds in case things go south.

(For the record, things went south and I got locked out nevertheless. Twice. Stay tuned as I will make copious fun of myself in an upcoming post, together with providing a few survival tips to recover a deployment gone belly-up.)

The final somewhat annoying bit of having a somewhat ephemeral public IP address is that you need to be careful with naked domains. I’m not a fan of naked domains per se, but at the same time I understand there is a general expectation that works just as well as so I will be going with the flow.

Short of the DNS powers that be finally realizing that it’s 2015 and the limitation on CNAMEs is something they should work on, the only solution is to bite the bullet and use the public IP Azure is giving you. I could have bought a few IP addresses for a very reasonable amount well within my MSDN subscription, butI care about IPv4 space so I didn’t feel my blog should squat one of the few left. Then again, while this IP may change, it is guaranteed to remain the same throughout the lifetime of the deployment and in my case I have no plans of flip-flopping this machine at all. I made a mental note to update the DNS in case I ever start from scratch and I got on with my life.

With these quirks out of the way I was finally able to wade my way into the world of FreeBSD jails: a topic for a follow-up post.

Hello from FreeBSD and Azure

This blog is now running on a set of FreeBSD jails on Azure. Setting it up was relatively easy and smooth, modulo a few bumps on the road that I have documented and will share in upcoming posts.

Why FreeBSD and why jails, you may ask? Everyone and their dog seems to be running Docker containers on Linux these days, so going FreeBSD can legitimately raise a few eyebrows. I wish I had a compelling answer and I could start pontificating about performance, security, scalability and the like but this is not the case: my blog doesn’t get nearly enough traffic to even start discussing scalability and I’m not nearly enough conversant on system internals to provide guidance one way or the other.

Truth is, I just resonate with FreeBSD more than I do with Linux. I cut my UNIX teeth on Linux, but I quickly moved to FreeBSD after my Linux workstation was pwned some 15 years ago. I came to FreeBSD for PF and stayed for make world.

Back when Linux was busy creating amazing things and paying a price in terms of slight yet annoying incompatibilities and quirks, FreeBSD has always been a cornerstone of predictable, solid performance. And usability too: hier is a joy, and knowing that anything that is not part of core goes in /usr/local and that port maintainers will leave configuration files alone (Debian, I’m looking at you and your a-little-too-clever Apache’s sites-available) takes a lot of guesswork away.

Over the years I deployed on Linux a number of times, mostly because I needed JVMs and as there wasn’t just enough choice of FreeBSD VMs out there. I did however make a point of being a pain in the backside of many colleagues at Microsoft until we got FreeBSD on VM Depot. The cherry on the cake was some downtime over the recent holidays which allowed me to learn my way around jails and successfully migrate my aging Linux machine over.

You are now connecting to FreeBSD 10.1, with four different jails running a DNS server, a MariaDB service and a couple of websites running mostly WordPress. A testament to FreeBSD quality is that to set it up I just had to dust some memory shelves and learn a couple of new things: despite not having deployed anything serious in the best part of 10 years, things are pretty much the same as they used to be, and the utmost predictability of what’s going to happen on a FreeBSD system is still very much there.

Please don’t get me wrong: it’s not like I don’t like Linux – I obviously did and still do although with many reservations on systemd. It’s just that one way or the other I keep coming back to FreeBSD: home sweet home.

So where have I been?

TL;DR – here:

That was a lot of miles!

Have open source, will travel. And travel more than what the map shows, as I have been multiple times in many of those countries.

Thing is, I haven’t stayed long. With the exception of China, where I usually stay for a whole week, I try to jam-pack as much as I can in my travels, and that usually means I’m in a country for an average of 36 hours before I move on. The craziest ones are the “three strikes” days: wake up early in country #1, fly to country #2, fly out in the late evening and sleepcrash in country #3. My back still holds a grudge for doing Sweden, Finland and Portugal this way.

As counterintuitive as it may seem, this travel pattern while positively crazy is incredibly productive. While I started to travel this way so that I could minimize time away from my family, I quickly understood it helps immensely to focus and get things done effectively. It does take a toll on planning, but there is a whole lot of things you can do in one day: my daily fare typically includes at least a few internal meetings, a couple of customer/partner visits, the occasional press roundtable and a talk at a community event.

It is frantic indeed, and sightseeing is pretty much out of the question unless it’s a drive-by on the road to or from a meeting. I do however spend lots of time with local colleagues, and that alone helps a lot understanding the place I’m visiting: many tourists may spend weeks in the same place, visit every corner, and yet fail to understand what a country is all about as they lack contact with local people. Me, I will forever cherish the two hours I spent last Monday with 40 students from Egypt, learning so many things about their country.

Another great way to learn about a foreign place is food. It takes some convincing, especially in far away countries, to talk colleagues into ditching the typical 5-star hotel international buffet for a local dive place, but it invariably pays off. I have very fond memories of a Padang place in Indonesia, and I will positively refuse to leave Singapore without a pilgrimage to the Maxwell Road hawker stalls. During this very last trip I discovered Ful Medames and Mulukhiiyah, and I am invariably looking forward to what China will have to offer in my next trip. For the germophobics reading this, know that in four years I have only been sick once, and that was when I couldn’t bail from a lunch at an Italian place in Beijing.

Then again, being with local people makes all the difference. And having a job where I get to meet so many local communities takes all the wear of travel away. I am sitting in my last hotel room in South Africa as I write this: I landed at 5.30am, gave an opening speech at an open source event at 9am, rushed to the office for a PR roundtable, shot a video and had a few internal sync meetings. As I try to pick myself up for my last day on the road, and then finally go back to my family I can’t help but thinking how lucky I am.

A tale of two countries

It boils down to shoes.

A lot of people asked me what it feels like to be an Italian in the USA. Books have been written, pictures have been taken and videos have been shot, so I feel I should spare you the details on getting Italian ingredients (you can get everything here), wrapping our heads about pounds and gallons (seriously, what were you guys thinking?) or remembering to have a full tank before hitting the outdoors (and a bear-proof food container if you plan to camp). Also, we are talking about a country that spans several time zones, so if there is no point trying to lump things up.

I do however have a story for you and I believe it pretty much sums up our experience in the Pacific Northwest, the corner of the States we have been lucky to dwell for the past four years.

A few months before leaving Italy, I needed new shoes. Nothing fancy, just an easy pair of loafers for the summer. I went to a shopping mall nearby, found what I wanted in a mid-level shoe store, paid up and off I was on my merry way. After a couple of days my new shoes split wide open, a tragic structural failure that had me limping home and later to the store to voice my disappointment.

They asked to see the receipt. They wondered if I did something weird with the shoes (though I don’t exactly look like I’m into parkour), they had me request to speak with the manager and finally they offered to repair, not reimburse – not replace, the offending shoe. They took their good time and after a few weeks I picked up a hack job of hot glue and amateur stitching that of course found its way to the trash bin shortly thereafter.

Fast forward a year or so, when I found another pair of loafers in a middle-of-the-road outlet store in North Bend. I liked them, tried them on, bought them and off I went. In a few weeks a small elastic band ripped off: it didn’t impair functionality, it was mostly cosmetic and a minor issue altogether. I didn’t bother going back to the store and actually kept that particular pair as my go-to shoes for quite a while. As a matter of fact, I still had those shoes on when I was back at the very same store almost a year later.

When the store clerk approached me, I told him I really liked the shoes I bought a year before, and I was wondering if they had something similar. He took a look at my feet and immediately asked what happened as he noticed the small rip in the fabric. I told him it was no biggie at all, but he insisted on seeing the shoe, asked when I bought it and told me in no uncertain terms that he was going to replace it under warranty. I told him I didn’t have the receipt, and he just asked for the credit card I used for the original transaction.

Minutes later I was driving home in my new free shoes, both me and my wife stunned at what just happened and commenting how good business sense, trust and, more generally, a positive attitude can make all the difference in the world. We love this place.

And so it goes…

You stop updating your blog. A few weeks go by, then months, then all of a sudden you realize it’s been years since you wrote anything. Starting again is hard, as you feel you should write something important, explain why you have been away, condense years of stories into a big catching up post, commit to a New Year’s resolution of getting back to blogging and try to stick to it – the works.

Or, you can just rediscover how much you really like FreeBSD, play with jails for a while, spend some time migrating a couple of databases and thinking that after all it’s just a blog, the world keeps on rolling with or without it and what is four years without posting among friends?

So here is to a fresh install of WordPress on the latest FreeBSD, all running on Azure (more in upcoming posts). And to whatever may come next. I’m not promising it will last, but it feels good to be back.

Sustainable software? Look down under!

A few months ago I was sipping a drink with friends, and I was asked what would I do should I ever leave Sourcesense. I answered that I would hope I’d make enough money by then but assuming it wasn’t the case, I would most likely start a new company or, failing that, I would contemplate moving to Sidney and send my CV to Atlassian.

There is more than surfing Australian waves in my admiration for that company: I’m watching with great amusement the debate on Open Source sustainability, how making money is tied to proprietary extensions, how Open Source is not a business model, and all the yadda-yadda that regularly pops in when someone dares to comment how, really, the Emperor is not wearing any clothes. Such commentaries are being filed in the “Firm grasp of the obvious” category, but they make for a fun read anyways: meanwhile, as the Commercial Open Source world is out there frantically looking for the Holy Grail of software sustainability in an open and collaborative ecosystem, it seems to me that a happy bunch of Aussies are filling it with Foster’s and passing it along.

While most Open Source companies try to make money by providing a free all-you-can-eat Sunday roast buffet, as long as you carve it yourself and bring your own gravvy, Atlassian is showing the beef by providing great food at reasonable prices, all the gravvy you want and a tab with no hidden charges, surprises or discretionary service fees attached. Not to mention a recipe book and access to the grill to cook to your own taste. Can you really argue with that?

I know, I know: it’s not Open Source, you need to pay to play and the ball is theirs. Yet their model is so upfront and clear that it feels like a breath of fresh air when compared to the amazing lot of commercial Open Source/crippleware in disguise out there:

  • pricing is clear and reasonable, mesured on real value instead than on what it takes to send a salesman to your premises to measure your spending ability, then provide you with a quote.
  • you pay for what actually drives value. Do you have 50 developers with software installed on their machines to build and test locally, plus a build and a staging server? No problem, here goes your unlimited free development license key to go along with the one you purchased for your production server.
  • do you want to tinker with the source code? You get all you need and then some to fix stuff yourself. And no, they won’t withdraw support just because you messed up with the code.
  • do you fancy ecosystems? Just browse the amazing number of plug-ins, add-ons and extensions that have been built by developers all around the world, or just ask for assistance in the user forums.
  • do you want to use their technology to support your Open Source effort? Here, get a free license and have fun. Oh, and by the way have a look at the notable number of contributions that Atlassian did to Open Source software and libraries they are using.

Can your Open Source vendor do this? I will need a few more fingers (and toes!) than I have access to if I wanted to count how many quote-Open Source-end-quote companies out there are doing their best to play the baitware game, providing astonishly little value for amazingly high prices and playing hardball with customers. While the Commercial Open Source world is talking about hybrid revenue models, here comes a pragmatic shop that just nails it. May I suggest analysts to pick up the phone and give Mike Cannon-Brookes a call?

The obligatory first post

Who am I not to oblige to write a quick post to test both my upgraded WordPress install and my brand new iPhone? This also works as an excuse to test the quite nifty WordPress iPhone app.
More later from a real keyboard, just bear with me while I turn a useless post into an hopelessly pointless waste of bandwith by adding a quick picture snapped with the phone camera outside the London house…


“Turn left at the bears, you’ll see elephants on your right”

This is the very first phrase I heard when me, Simone, and a truck full of beverages arrived to the location for this year’s Cocoon GetTogether. Such things happen if your conference venue is actually a zoo.

From then on, it has been an incredible roller coaster: you really don’t know what “busy, crazy busy” means until you have organized a conference.We have been through an amazing three days stretch with a great bunch of old friends and a surprising number of newcomers: the Cocoon community never ceases to amaze me: for the fifth time in a row we managed to get an incredible group of people, traveling from all over the world (even from Australia!) to enjoy the company of fellow Cocoon hackers and share some nice food and drink in sunny Rome.

I’m expecting to be in tatters for the next week or so, but I’m really, really enthusiastic about the conference so far. We had a productive hackathon, a couple of fantastic evening events and, of course, a conference packed full of great information about the latest and greatest in Cocoon-land (if you didn’t check the new Cocoon website, this is an excellent time to do so). I’m really lost for words when it comes to describing what we’ve been up to so far: just know that by missing the conference you’ve missed a chance to understand how loud an elephant trumpeting a tiny wall away from you can be. Or how you can eat humongous quantities of food, drink enormous quantities of wine and sing along at a table full of hackers in the Roman countryside. And this isn’t even scraping the surface of what the CocoonGT has been: you can probably get a small feeling of what we’ve been up to by looking at the photos popping up on Flickr, but that’s really not going to give you the full picture. You have to be here to understand how the Cocoon community is different, so stay tuned and don’t miss the sixth edition!

The Futurama effect

Sometimes there is a lot of stuff we take for granted. Being somewhat an Internet-based professional kinda takes all the magic out of this amazing connected world and the changes that have been happening among us. Imagine a guy being hibernated some twenty years ago and waking up today. Try to figure out the astonishment and the overwhelming sensation of a world that has taken a quantum leap since he’s been refrigerated. Try to explain to him stuff like the Internet, the World Wide Web, Wikipedia, social networks and the like: chances are you’re in for a lot of blank stares and puzzled looks.

This is what just happened to me.

See, while on vacation some neurons of yours truly had a somewhat funky connection, and I figured out that I really want to play piano again, something I dropped in my teenage years. I distinctly remember how it felt back in those days: I was a broke teenager living in a small town, relying on pocket money and pub jobs to earn my fun. Living in a small place meant little to no access to information, and being nearly penniless didn’t quite help. I had to travel 40 miles to get to an halfway decent sheet music store, I wasn’t in touch with other pianists, (vinyl) records were too expensive for a teenage and I was frantically videotaping anything that vaguely resembled classical music performances on our mere six national channels TV. I still remember staying up late to record the Bach Series from Glenn Gould, and I clearly recall saving a few pennies here and there to finally buy a copy of Schubert Impromptus. Later on, I got sucked into different hobbies and life in general, so I basically dropped my piano altogether.

Fast forward twenty years later. Here is a guy sailing towards his forties who has been fortunate enough to witness almost all the Internet stuff happening in his country, from SLIP and UUCP access to ADSL2 and beyond, from NCSA servers to IPTV, from Swish and Altavista to Google. I have been fortunate enough to work behind the scenes and somewhat help making all this happen: it’s a great ride, but with the downside of spoiling the magic. Gone are the times where I stared at my 386 receiving a ping response from the other side of the Atlantic. I got used to the Internet in small steps, and to the idea that information is at my fingertips, be it a phone number, travel directions or software being written cooperatively by a community of strangers living all around the world. All this happened gradually, and it just became part of my life: I’m just expecting to browse for information and finding it somewhat.

The teenage in me just woke up, though, and wants to play piano. He finds himself in front of a computer screen, and understands that instead of relying on a lousy local shop, his piano might be coming from 700km away. He doesn’t have to take the vendor’s word, as there are a plethora of forums and sites full of qualified reviews. He can even drop an email to a good friend in the UK and get some great advice. There might be no need to buy music sheet as  no less than 12.618 scores are available for free download. Want to learn from pros such as Horowitz, Richter, Pollini or Ashkenazy? YouTube has plenty of that. Piano lessons? No brainer, there are instructors from Europe, USA and Japan sharing material on the Net. Care to share some thought with people with similar interests? Go figure the sheer number of piano communities out there.

This is just overwhelming, and so different from what it used to be. It’s a weird sensation being exposed all at once to such an unexpected massive amount of valuable information, and it really made me wonder. Most of us just don’t realize how exciting these times are and how lucky are we to witness the quantum leap the world has been through in just a handful of years.

Sometimes it’s really good to stop for a moment. And think about it.

See you in Philly!

I just got home from the Open Source Think Tank, and it’s almost time to pack again and cross the Atlantic one more time. Next week I will be in Philadelphia, talking about Open Source in Corporate Environments at Emerging Technologies for the Enterprise.

As my schedule gets tighter, this time I will have no room to hang out and have a look around: I will be landing in Tuesday and fly back on Thursday evening, which will make jet lag interesting stuff to master. If you happen to be around the Philadelphia area, just wave and I’ll be glad to have a chat over a beer or so while I’m attending the conference.

Hopefully travel won’t be too bad: I decided this was a good occasion to jump the cliff and give the “all-business” airlines a chance, so I’ll be flying via Paris with L’Avion, spending 70% than the coach class rip-off for short-length trips and enjoy a better seat, possibly with some sleep included on my way back. Bonus treat: I will escape the Heathrow security jokes. As it seems I’ll be crossing the ocean a number of times in the future, I hope this proves to be a good compromise between sustainable prices and travel comfort: where do I sign to have those guys open routes to the West Coast?