Anecdotes About the macOS Sandbox File Limit

As always, this post reflects my own opinions, and not anyone else’s.

There has recently been some renewed discussion on Twitter about the limitation that the macOS sandbox places on the number of open files an app can access at once. Apps are still running into problems because of this limitation, and there is not a lot of technical detail available on it outside of Apple, so I’d like to share my understanding of it.

macOS only allows sandboxed apps to access a limited number of files at a time, but there is no way for an app to query how many files it can open, or if it’s close to the limit. In fact, this limit is dependent on the amount of RAM installed in the computer and the number of files open by other apps. Because this limit only affects apps that can batch process thousands of files, and users that want to do so, many users—and developers—remain unaware of it, despite the fact that it has affected some of Apple’s own apps.

I first became aware of the limitation in early 2012. I was working on an app that could activate font files, and although most users only used it for a handful of fonts at a time, we very quickly got support requests from users after the limitation was introduced in an update to OS X 10.7. In this case, however, it wasn’t obvious what was causing the issue. Font activation would start failing with a generic error code after the first few hundred activations. The system log reported that the sandbox was denying file-read-data for our app, but our app wasn’t sandboxed, so this appeared to be a bug.

Although we attempted to fix the problem, both by ourselves and with the help of Apple’s Developer Tech Support, it wasn’t until WWDC 2012, a few months later, that we were able to get answers on why this was happening. At WWDC, I lined up for the labs as soon as I could and was eventually able to talk with an engineer who worked on font activation. After sharing some code with him that demonstrated the problem, he was able to find exactly where in the font daemon the failure was occurring. When attempting to open the font file, the daemon was receiving a generic error that the file could not be opened. However, he wasn’t sure exactly who to talk to about that problem, so he gave be a page of detailed notes, along with suggestions of other engineers to talk to, and which labs they would be in.

All of the engineers were helpful and wanted to get to the bottom of this mystery, but they all had to suggest other engineers to speak with, until ultimately, I was referred to an engineer in the security lab. It was late in the day, and labs were about to close. I distinctly remember running across the room to the section blocked off for the security lab, laptop under my arm. There, I met an extremely helpful engineer who was able to solve the mystery.

macOS uses security-scoped bookmarks to grant apps access to files they normally wouldn’t have access to. In order to facilitate this, macOS must keep a mapping of which apps are allowed to access which file paths, and for security reasons, this mapping has to live in kernel memory. Kernel memory is wired: The kernel has no access to virtual memory. A quarter of physical RAM is reserved for the kernel, and within that quarter, the kernel allocates a percentage for the file mapping.

He never said it outright, but I got the impression that the security team had just learned that this was causing problems for apps recently, and only because they were notified by the iPhotos team, but they were unaware it was causing problems with the font daemon. Our radars, DTS tickets, and emails with our Apple rep had failed to get the problem through to the security team. As an aside, this is why WWDC labs are so important and valuable. It is often impossible to get issues noticed by the proper engineers inside Apple. Even if they can’t fix the issues, engineers in the labs can often provide information and workarounds.

The engineer stayed late to help me file radars on the issue for both the security team and the fonts team, but cautioned that the underlying problem would probably not get fixed any time soon. It would be up to app developers and other teams within Apple to work around the problems the limitation created. It was foundational to how the macOS app sandbox was designed, and could only be fixed if they found a safe and performant way to store this mapping in userspace, which was unlikely. Apple was not going to throw out the sandbox or compromise its security to remove this limitation. Considering that it remains a problem to this day, he was right.

From outside of Apple, it appears that very little has changed in the app sandbox since its debut in 2011. Many exception entitlements are still “temporary”. When these temporary exceptions were introduced, Apple encouraged developers to file radars explaining why they needed the exceptions, indicating that Apple would, in the future, expand the sandbox to cover use cases that necessitated these exceptions, then remove the temporary entitlements. As far as I am aware, although security holes have been patched in the sandbox, and new entitlements added, the sandbox has never been expanded to include these use cases, and no temporary exception has ever been removed, making “temporary” somewhat of a misnomer.

Having never worked at Apple, I will refrain from uneducated guesses as to why this is. Considering that this limitation still affects high-profile apps like Microsoft Office, I don’t think Apple is unmotivated to fix this, but considering how long it has remained a problem, until Apple announces otherwise, we can’t assume it’s going away any time soon. It’s on app developers to work around it as best as possible.

There are two ways to deal with the issue. The first is to relinquish security-scoped bookmarks as quickly as possible, and the second is to prompt users to open folders, not files.

Security-scoped bookmarks can be relinquished using NSURL's stopAccessingSecurityScopedResource method. This method frees the entry for the file in the kernel. If your app only needs to access a few files at a time, you can call this method immediately when you are done with each file.

In cases where this does not work, you can prompt users to open folders instead of files. When a folder is opened in an NSOpenPanel, only one entry is added in kernel memory, but your app will still be able to access the files within that folder. However, you may need to limit yourself to APIs that access those files using string paths, rather than NSURLs. After opening the folder, your app will need to enumerate the contents of the folder and provide its own file listing for users to select files. Any files opened through NSOpenPanel will consume kernel memory.

In my case, thanks to the help I received at WWDC 2012, I was able to learn that the font daemon properly relinquished bookmarks when fonts were activated using the CoreText API, as opposed to the newly-deprecated ATS API we were using. We were able to switch over to CoreText, but Carbon apps, which all used ATS, wouldn’t be able to see all of the activated fonts. It appeared that these apps, upon being notified of font activations, would communicate with the font daemon to get the font data using ATS functions, which were still not relinquishing the bookmarks quickly enough. As far as I know, this has never been fixed, but since Carbon was never fully ported to 64-bit, Carbon app usage declined until 32-bit support was removed in macOS 10.15, making the problem moot.

At the time, however, many popular apps were written using Carbon, or at least contained enough Carbon code to be affected by this issue, including Microsoft Office and parts of the Adobe Creative Suite. We had to add an alert to our app when users attempted to activate a large number of fonts, explaining that not all fonts would be available in these apps. Some users complained bitterly, but we had a compelling demo to prove that the problem was not our app’s fault. If users simply moved all their font files into ~/Library/Fonts, they could confirm that the fonts were all available in TextEdit, but not in Word.