macOS Catalina has wrecked my home tech setup – UPDATE

UPDATE: These issues also extend to WebDAV file sharing. So in macOS Catalina there is no usable file sharing mechanism (from a server perspective) that does not have severe defects. Sadly the Catalina 10.15.3 update does not appear to have addressed any of these issues…

My family and I are long time, very committed, users of Apple products and the associated eco-system. We switched to Apple from Microsoft back in 2010 and until now we have mostly been happy, apart from the occasional issue.

Unfortunately this has all changed with the release of macOS Catalina; Apple has utterly broken critical core functionality within macOS that we have relied on for nearly 10 years. The issues are clearly very serious bugs yet I get no acknowledgement of this from Apple (I have logged multiple bugs pin all of the issues). Let me describe the (multiple) issues…


Apple systems can share folders with other systems using File Sharing. There are three main protocols that can be used for this.

SMB: (Server Message Block) This is the most prevalent file sharing protocol. It is a Microsoft protocol which has been used by Windows for a great many years. It is also used on Linux via the open source Samba software. It is pretty much an industry standard nowadays.

AFP: (Apple File Protocol) This is a common sharing mechanism within the Apple eco-system. It is a proprietary sharing mechanism (though various third parties also support it) and for a long time was the primary file sharing protocol for Macs. A few years ago Apple declared their intent to phase out AFP in favour of SMB.

NFS: (Network File System) This is main Unix/Linux file sharing mechanism. It is an industry standard that has been around for many, many years. It works well but has some idiosyncrasies and limitations for day-to-day file sharing in macOS and Windows environments.

In our family we have a total of 9 Macs, 5 iPhones, 5 iPads, Apple TVS and Apple Watches. We use Apple services such as iCloud, Apple Music etc., extensively.

For many years (at least 6 or 7) we have had a setup at home whereby we have two Mac systems that act as ‘servers’. They make our central storage accessible to other Macs via file sharing and also provide other services, such as video streaming, to our Macs and mobile devices, both locally and via the Internet. These servers and services operate 24×7.

Our primary storage server is a Mac mini running Mojave 10.14.6. Connected to this via a Thunderbolt connection is a Drobo 5D3 with 50 TB of raw storage formatted as HFS+. This storage is mounted locally on the Mac mini as /Volumes/NAS (the name is historical – this is directly attached Thunderbolt storage). This volume is shared with clients via SMB, AFP and NFS.

Our secondary storage server is another Mac mini which is now running Catalina 10.15.2. It shares an external (USB attached) volume, /Volumes/Music via SMB and AFP.

Our 7 Mac ‘clients’ (all currently running Catalina 10.15.2) are a mix of iMacs and MacBook Pros. They all mount these shared volumes using either SMB or AFP and have the shared volumes  permanently mounted. The clients are rarely restarted (once every few weeks maybe) but do routinely go to sleep and wake up. In particular they typically sleep for several hours at night (dark wake and power nap notwithstanding) except for a couple of hours when they wake to perform (non Time Machine) backup tasks.

The network used by these systems is our wired home network. This is a GigaBit network connected via CAT-6 cabling and provided by an enterprise grade L3 managed switch (D-Link DGS 1210-48). Our WiFi network (which is not really involved in this issue) is provided by three Apple Airport extremes (final generation ‘tower’ 802.11ac models); one acts as our router and all three act as WiFi access points. The WiFi network is extended via VLAN using Apple’s extend the WiFi network over the wired network capability. DHCP is provided by my two Synology NAS units (operating in a redundant configuration), local DNS services are also provided by my two Synology NAS units (so I have redundant DNS capability) and external DNS is provided by CloudFlare ( Our home network is full dual-stack IPv4/IPv6 as is our Internet connection. This is a fast and very reliable network.

Both my Synology NAS units are time-synced via NTP to and act as local time servers for my network. All the local Macs are time-sync’d to the Synology NAS units.

I have comprehensive monitoring and alerting in place so if any aspect of this setup experiences an issue I know about it very quickly and in many cases the issue will be automatically corrected.

This setup has been operating very reliably for many years, until that is I upgraded to Catalina (and more specifically 10.15.2). Since the upgrade my file sharing setup has been horribly broken due to three serious issues which I describe below in order of decreasing severity.

1. SMB causes clients to hang, requiring a hard power off to recover

All of the Macs that mount shares from the servers via SMB experience hangs related to SMB file sharing. When they wake from sleep in the morning one or more of the mounted SMB shares is ‘hung’ Any attempt to access the share (even indirectly) causes Finder to beach-ball and shortly after that the rest of the system starts to become unresponsive. It is not possible to perform a clean restart (the system does not respond) so I have to forcibly power cycle the Mac. This happens on every client Mac (all 7) every day. We never experienced this issue prior to 10.15.2 being installed on the client Macs.

I know this is SMB file sharing related because I have switched all of the Mac clients to mount the shares via AFP file sharing and the problems have totally disappeared. Everything is now 100% reliable again. However AFP is deprecated and is not a long term solution, especially given (2) below…

2.  AFP file sharing is broken in Catalina

In Catalina Apple has crippled the AFP server. If you share an external volume, or a folder on an external volume (or indeed anything mounted under /Volumes) from a Catalina server, then when a client mounts that folder via AFP one of two problems occurs:

a)   If the client is running Mojave, the folder appears to mount correctly but the client cannot actually see anything within the mounted are (it appears empty in Finder and from the command line). Any attempt to write to the folder results in a ‘permission’ denied message. If the client mounts the same folder via SMB then everything works just fine (except for issue (1) above of course).

b)  If the client is running Catalina then the share does not even mount properly; the mounted folder gets assigned incorrect permissions (d–x——) and so is not even accessible from the client. Again if the share is mounted using SMB then it works fine (apart from (1) above).

This used to work just fine under Mojave and earlier versions so something in Catalina has broken this. Luckily one of my ‘servers; is still running Mojave so I am able to use AFP for our primary storage to avoid use (1) but our secondary server is now essentially crippled and we have essentially lost access to our shared Music catalogue. It also means that this server is stuck on Mojave until Apple decides to fix these horrendous bugs.

3.   NFS file sharing is impaired in Catalina

In Catalina, if NFS is used to ‘export’ (NFS terminology) a filesystem or folder residing under /Volumes (so that the filesystem or folder can be mounted by NFS clients) then the filesystem is not exported correctly. Specifically it is only exported vis IPv4 not via IPv6 even if IPv6 is specified. This issue only affects filesystems and folders on external devices; filesystems and folders on internal devices work as expected. Again this issue never existed prior to Catalina.


These are very serious bugs that break core macOS functionality. Catalina should never have been released while such bugs exist. Even more troubling is that some of them seem to have been introduced in the 10.15.2 update.

It’s interesting that issues (2) and (3) seem to only affect things mounted under /Volumes, which is a firmlink. My guess is that these issues may be due to incorrect handling of firmlinks by the AFP and NFS servers and/or the new privacy protection features in Catalina. Either way Apple needs to get its finger out and fix all of these issues post haste!

Posted in IT and Computing | 4 Comments

Avoiding issues with Netflix when using an IPv6 tunnel: Update

Following the migration of my home DNS infrastructure from a mixed macOS Server / Synology DNS Server setup to a Synology only setup (still with dual redundancy) it has become easier to avoid the Netflix IPv6 issues.

The Synology DNS server supports the notion of ‘forward only’ zones (macOS Server DNS kind of supported these but they were not officially supported and the Server DNS GUI didn’t provide any way to configure them).

My new setup is as follows:

I run my modified DNSmasq build on my two macOS Servers configured to return only IPv4 addresses for and all sub-domains.

My primary Synology DNS servers both have defined as a ‘forward only’ zone forwarding to the macOS hosted DNSmasq servers.

This allows me to retain a single cohesive DNS setup for all systems (they point just at the primary Synology servers) and have full Netflix capability on any device that needs it.

Posted in IT and Computing | 3 Comments

Replacing macOS Server

Until recently I used macOS Server to provide numerous IT related services for my home network and my users (my family). For example:

  • User authentication (Open Directory)
  • VPN server
  • Calendar and Contacts
  • Email
  • Wiki
  • Websites (including this WordPress site)
  • DNS
  • DHCP
  • File sharing (SMB and WebDAV)

When Apple announced that they were changing the focus of macOS Server and that many of these functions would be deprecated and then removed I decided to waste no time in identifying replacements and migrating to them. This post covers what I used to replace each of these services…

As well as macOS sServer I also had a Synology NAS unit (DS1812+) and this supports many different applications. I had anyway planned to augment this with a second unit to increase my backup storage capacity and so it seemed like this might be the way to go, and indeed it was.

I augmented by existing DS1812+ with a new DS1817+ and then I migrated most of the macOS Server services to various Synology provided applications (all free) as follows:

VPN server -> Synology VPN server

DNS -> Synology DNS

DHCP -> Synology DHCP

Email -> Synology Mail Server + Synology Mail Station (WebMail)

WebSites -> Synology Web Station

WordPress -> WordPress hosted on Synology

Wiki -> WordPress hosted on Synology

For Calendar and Contacts I retained an Apple based solution but used the iCloud option instead of hosting them myself:

Calendar -> iCloud Calendar

Contacts -> iCloud Contacts

I abandoned the notion of a central user directory and single sign on; it is unnecessarily complex for a home environment. Instead I wrote some custom tools to allow for easy password change / sync across all of the different platforms now in use (macOS, Synology, Apache, WordPress).

I retained macOS as my file sharing solution (SMB and WebDAV) since my macOS Server hosts my Drobo 5D3 Thunderbolt connected disk array.

The migration process was surprisingly easy and we have not lost any significant functionality as a result. Having two Synology units has actually allowed me to increase the level of redundancy for my services; now all of my self hosted services have dual redundancy for increased reliability and resilience to outages. The overall level of data protection has also been enhanced.

All in all I’m very pleased with how things have turned out, though I am still extremely disappointed with Apple’s decision to essentially kill macOS Server.

Posted in IT and Computing | Leave a comment

Avoiding issues with Netflix when using an IPv6 tunnel

As you know from my previous blog posts, currently my home network connects to the Internet via Virgin Media, who currently does not offer native IPv6 connectivity. To provide IPv6 Internet connectivity for my home I use the excellent service from Hurricane Electric. Unfortunately there is a significant issue with this setup; Netflix does not work!

The reason for this is because (a) most systems nowadays prefer IPv6 over IPv4 if both are available, (b) Netflix servers are available via public IPv6 addresses and (c) Netflix checks IPv6 connections and blocks ones coming via tunnel providers, VPNs, proxy services etc. This is part of their attempts to geo-fence content, driven primarily by content provider demands.

I’m not going to get into the right and wrongs of this blocking but I am going to present a relatively easy (and perfectly legitimate) way to avoid the issue. Basically, what I want to do is to constrain all traffic to Netflix from my home network to be IPv4 only (so it will be native connectivity via my ISP) while not impeding normal (mixed IPv6 and IPv4) traffic to anything else. There are several ways that you might do this but I wanted one that was flexible, which could be applied to multiple devices in my home easily and which was transparent/non-disruptive to the rest of my home network and connected devices.

The solution I settled for was as follows;

  1. I took the latest version (2.77) of the popular lightweight DNSMASQ DNS/DHCP/TFTP server and added an enhancement to it to create a new version (2.77cj). With this enhancement you can specify a new parameter ‘v4only-file’. The value for this parameter is the pathname of a text file containing, one per line, host or domain names that you only want to receive IPv4 addresses for in response to DNS requests (i.e. IPv6 addresses – AAAA requests – are filtered). The matching is somewhat smart in that it matches the end of the name in the DNS request against the entries in the file so you have a leading ‘wildcard’ effect.
  2. Run this modified DNSMASQ as a ‘forward only’ DNS server on a server in my home network and use my existing two DNS servers as the forwarders (so that requests to DNSMASQ will resolve machines on my home network not just those on the Internet).
  3. For any machines that want to use Netflix (primarily just a couple of iPads and one Mac), set their DNS servers to point (only) to the machine running DNSMASQ.

With this setup and with just one line, ‘netflix’com’, in the filter file Netflix works just fine and there is no impact on any other functionality.

Hopefully Virgin Media will get their act together and roll out native IPv6 connectivity soon and then there will be no need for this ‘hack’.

Posted in IT and Computing | 3 Comments

IPv6 at home

Having recently moved my home infrastructure to a primarily Apple OS X Server / Synology NAS base (see my previous posts), I decided to investigate the practicalities of deploying a full IPv6 / IPv4 co-existence setup on my home network and maybe even to enable IPv6 for Internet access as well. I was not sure how feasible this would be or how difficult. I was amazed at how easy it was!

Firstly, virtually every piece of equipment and OS that we use at home seems to be fully IPv6 capable:

  •  Apple Airport Extreme Wifi base station and router
  • OS X
  • OS X Server services (apart from the VPN server)
  • Synology NAS, including DNS server
  • iOS 7 on iPhone, iPad and Apple TV
  • Windows 7

 The next question was how easy it might be to get actual IPv6 Internet access. My ISP (Virgin Media) is not yet natively deploying IPv6 but a quick search revealed TunnelBroker from Hurricane Electric. This allows you to setup a (free) account and then create one or more IPv6 over IPv4 tunnels to allow your home IPv6 network to access the IPv6 Internet over an IPv4 connection. The site also provides a lot of useful information on IPv6 in general, how to configure it on many different OS and its current level of adoption across the Internet. It was quite surprising to me to see how many web sites and companies already have an IPv6 presence on the Internet. Among the top names are Google and Wikipedia.

So, having created my tunnel the next thing was to configure the Airport Extreme router to use it. I anticipated that this might be difficult or complex but it was in fact very simple, almost scarily so, by just following the comprehensive information provided on the TunnelBroker web site. It is recommended that you have the latest Airport router firmware, which I already did.

At this point I should mention a very important thing about IPv6 and the Internet. With good old IPv4, your router typically implements NAT which essentially ‘hides’ your home network from the Internet and makes it much harder for any of the nasty things lurking in the darker corners of the Internet to invade your home network. It also makes it more complex to expose services (web sites, e-mail servers etc.) on the Internet but that is generally considered a price worth paying for the protection. This is not the case with IPv6.; the whole idea behind IPv6 is that all devices should be visible on, and accessible from, the Internet by default; there is deliberately no concept of NAT. However, any router that supports IPv6 should provide an IPv6 firewall function. You should be very sure to turn this on and configure it suitably to avoid unwelcome ‘visitors’ to your home network. The Apple Airport Extreme has a comprehensive IPv6 firewall so I enabled this and setup a rule to only expose our public web-site via IPv6. For now I am leaving all our the services that we expose via IPv4 (mail, calendar, contacts etc.) fire walled even though they use SSL, require authentication etc. As use of IPv6 becomes more common over the next few years I will open those up too.

Now that we were connected to the IPv6 Internet the next thing was to setup the home network. IPv6 has an ‘auto configuration’ mechanism and this works very well. The router is responsible for assigning fully routable (i.e. public) IPv6 addresses to every device that asks for one. Due to the way that IPv6 address allocation works, each device on the home network will always get the same public address allocated for each interface (IPv6 addresses are assigned to interfaces not hosts). This made it easy to add all the necessary IPv6 addresses into my home DNS setup so machines could easily find out each others IPv6 addresses and talk to each other via IPv6. I was pleased to discover that the OS X Server DNS server and the DNS server in the Synology NAS both fully support IPv6. Not only do they support adding IPv6 addresses for hosts and defining IPv6 reverse zones but they also support DNS queries via IPv6. Cool!

Once all my main machines (Server, NAS, Mac and Windows clients) were setup to use IPv6, the next thing was to test it out! I embarked on a program to test all our internal services to see if they worked over IPv6. The great news is that they do! Here are the services that I have tested and confirmed to work over IPv6:

  • SMB/SMB2 file sharing to Apple Server / Synology NAS from Mac and Windows clients
  • AFP file sharing to Apple Server and Synology NAS from Mac and Windows clients
  • NFS file sharing to Apple Server and Synology NAS from Mac and Windows clients
  • OS X Server web access (HTTP and HTTPS) including Wiki service and Profile/Devcie Manager service
  • DNS (OS X Server and Synology NAS)
  • Caching (app store and software updates)
  • Calendar
  • Contacts
  • Mail
  • Time Machine
  • Open Directory

 So pretty much everything with the exception of VPN; but I’m not quite sure yet (more research needed) how VPN works in an IPv6 environment anyway!

Lastly I tried accessing a few of the IPv6 enabled web sites out there and was delighted to see that Safari (Mac),  FireFox (Mac and Windows) and Chrome (Mac and Windows) seem happy to use IPv6 if the web-site is accessible over it (I did not test Internet Explorer since I never use that).

I have to conclude that IPv6 seems very mature, much more so than I had imagined and I will be leaving my home setup configured for full dual-stack operation in readiness for when my ISP starts to support IPv6 natively. Until then I will continue with TunnelBroker.

Posted in IT and Computing | 2 Comments

Home IT: Part 3

Things have moved on a little since my last post. I have now retired the Active Directory infrastructure and the associated server VMs and the Windows 7 machine that hosted one of them (that should help our electricity bills a little bit!). My backup DNS and DHCP servers now run on the Synology NAS unit. I have local users defined on the NAS unit to control access to that storage from our PCs, Macs and iThings. Everything else uses Open Directory for authentication and authorisation. Even our Windows PCs can authenticate logons against OD using the free pGINA software which works very well.

So, although we still have a few Windows PCs, our infrastructure is now all OS X and Linux (i.e. Synology NAS) based. Which is nice.

Posted in IT and Computing | Leave a comment

Home IT: Part 2

For the best part of my working life I have been in the Microsoft camp, mainly by default. I started working with DOS 3.x back in the mists of time and have continued through Windows 2.0 all the way to Windows 7. As a result of this ,and having some spare licences available, when I was setting up a home network several years ago I went for a Microsoft based solution with Windows Server 2003 running Active Directory, IIS 6.0 for the web server and Exchange 2003 for the e-mail server. Kind of overkill for a home setup!

More recently (around 2010) I bought an iMac and then an iPhone and iPad. Suddenly I realised what I had been missing and now I am very much an Apple Mac devotee. We still have several Windows machines as well so my home setup has had to evolve to handle both. In order to reduce space and power requirements I recently rationalised my server setup and now it is as follows:

Synology NAS. As well as being our primary storage server the NAS unit also provides some other services. Most notably it runs a Logitech Media Server instance which serves up our music library to our two Logitech SqueezeBox Touch players.

Mac mini (Late 2012) Server with 2.6 GHz Core i7, 16 GB RAM, 256 GB SSD. This is running Mountain Lion Server and it provides the following services:



Open Directory

File Sharing


Mobile Device Management

Calendar Server

Contacts Server

E-mail Server

Web Server

Wiki & Blog Server

iTunes Home Sharing Server

Windows 7 ‘server’ 3.2 GHz Core i7, 16 GB RAM, 256 GB SSD, 2 x 2TB HDDs. This does not really run anything important any longer other than hosting one of the ‘virtual’ active directory servers.

Two Windows Server 2008 R2 servers. These run Active Directory and DNS. They are both running as virtual machines under VirtualBox, one hosted on the Mac mini server and the other on the Windows 7 server. They provide single-sign on and permission management for our several Windows machines and the family Mac. The Synology NAS is also bound to the domain so access permissions can be applied globally and consistently.

In the longer term I would like to phase out Active Directory completely but that depends on Synology providing better support for authentication and authorisation against Open Directory (which may never happen). Windows 7 can authenticate against Open Directory using the pGINA plugin so that is not an issue.

I have three Uninterruptible Power Supplies (UPS) to protect the servers, NAS, switch, router, cable modem etc. from power outages and spikes and to allow for a controlled shutdown in the event of a prolonged power failure.

Posted in IT and Computing | Leave a comment

Home IT: Part 1

I’ve decided that my blog will mostly focus on IT things and in particular ‘Home IT’. Some people might wonder what on earth ‘Home IT’ is and indeed for many people there is no such thing. Having said that, nowadays we all tend to have more and more gadgets (PCs, Macs, iThings, mobiles, tablets, media centres etc.) in our homes and also many more every day devices such as TVs and PVRs are now ‘connected’. So, the home really is becoming an ‘IT place’ for many people even if they do not realise it. This is fine until things go wrong 🙂

Being an IT / Software guy my home is probably more ‘IT’ than most so in this article (and some subsequent ones) I will give an overview of my home setup.

Our Internet connection is a Virgin Media broadband (cable) connection with max speed of 120 Mbit/s downstream and 10 Mbit/s upstream. It is pretty reliable and the typical speeds are also very good. In testing with I routinely see downstream speeds > 80 Mbit/s and getting the full 120 Mbit/s is not uncommon. Upstream is always in the 8 – 10 Mbit/s range. Overall no real complaints. The Cable Modem that I have is a Virgin Media SuperHub. This is a cable modem / router / WiFi access point made for VM by NetGear. I do not use the router or WiFi functions – I have the unit configured in ‘modem mode’ so that it acts just like a cable modem with no router or WiFi  functions.

My router, connected to the cable modem, is an Apple Airport Extreme. This is not the latest model (just released this year) that supports 802.11ac but the previous one that supports 802.11n/a/g/b at 5 GHz and 2.4 GHz. It is a very nice unit with great performance both as a router and over WiFi. Configuration is very easy via the Apple Airport utility which runs on Mac OS X,Windows and iOS. The unit is not quite as flexible, configuration wise, as some units I have had in the past but it does everything I need and is reliable and fast. No need to reboot this baby every month!

My house is partially cabled with CAT6 cabling (which we had installed during some major building work in 2007) so my core home network is a GigaBit wired network driven by a central 24-port layer 3 managed switch from D-Link. This is a professional level switch with a host of cool features and forms the backbone of the home LAN. There are a few other GigaBit switches deployed here and there around the house where necessary but most devices that use wired connections connect directly to the main GigaBit switch. This means I have a high performance wired LAN accessible from most places in the house.

As I mentioned earlier, my router is an Airport Extreme and this also provides WiFI access. However, in some parts of the house the signal from this is quite weak so I have augmented it with a second Airport Extreme at the other end of the house to extend the WiFi network. Both Airport Extremes are connected to the main GigaBit switch and so our WiFi coverage is both comprehensive and fast. Most of our WiFi devices support 802.11n at 5 GHz so we can easily get 100 Mbit/s or more WiFi connections anywhere in the house. Great for streaming video etc.! The sharp eyed might wonder why my second access point is an Airport Extreme (which also has router capability) rather than a plain Airport Express (which is just an access point). The reason for this is redundancy. If my main Airport Extreme should fail then I can quickly repurpose the second unit as a router until the original is repaired or replaced. As you will see in subsequent articles I take redundancy and service continuity quite seriously. You might wonder why but when you have a wife and two children complaining that the Internet is down or that they can’t watch a program off the Apple TV then you will understand why 🙂

For centralised storage, I have a Synology Network Attached Storage (NAS) unit, a DS1812+. This has 8 x 3 TB drives configured using Synology Hybrid RAID with dual disk redundancy; up to 2 of the drives can fail simultaneously and there will be no loss of data and the unit will continue to function. The failed disks can be replaced without shutting down the unit. This setup gives me 18 TB of usable raw storage which equates to around 16 TB after formatting etc. The NAS unit is really a Linux based server in its own right and as well as providing network storage accessible to Windows machines (SMB sharing), Macs (AFP sharing) and Linux (NFS sharing) you can also run many different services on it. The NAS is connected directly to the central GigaBit switch via dual GigaBit connections configured as a ‘bond’ providing both load balancing and redundancy.

The NAS unit is used to store the bulk of our data from important documents to our photo library to our media (music and video library). It is also used to provide network backup services (via Time Machine) to our three Macs. Since RAID  provides protection against drive failure but not against e.g. human error most of the contents of the NAS are also backed up to a directly attached (eSATA) RAID-5 array (5 x 3 TB drives) and also to a remote cloud storage service. These backups are managed by the excellent CrashPlan running natively on the Synology unit itself. So, we are reasonably well covered for protection and backup of our important data.

That’s all for this article. In the next one I will describe the various servers I run and what they are used for.

Posted in IT and Computing | 2 Comments

This is my blog

This is my blog. I doubt that I will post things here very often but when I do I will try and make then interesting.

Posted in Miscellaneous | Comments Off on This is my blog