Swift is cross-platform, but it behaves differently on Apple platforms vs. all other operating systems, mainly for two reasons:

• The Objective-C runtime is only available on Apple platforms.
• Foundation and the other core libraries have separate implementations for non-Apple OSes. This means some Foundation APIs may produce divergent results on macOS/iOS and Linux (though the stated goal is implementation parity), or they may simply not be fully implemented yet.

Therefore, when you write a library that doesn’t depend on any Apple-specific functionality, it’s a good idea to test your code on macOS/iOS and Linux.

Test discovery on Linux

Any unit testing framework must be able to find the tests it should run. On Apple platforms, the XCTest framework uses the Objective-C runtime to enumerate all test suites and their methods in a test target. Since the Objective-C runtime is not available on Linux and the Swift runtime currently lacks equivalent functionality, XCTest on Linux requires the developer to provide an explicit list of tests to run.

The allTests property

The way this works (by convention established by the Swift Package Manager) is that you add an additional property named allTests to each of your XCTestCase subclasses, which returns an array of test functions and their names. For example, a class containing a single test might look like this:

// Tests/BananaKitTests/BananaTests.swift
import XCTest
import BananaKit

class BananaTests: XCTestCase {
    static var allTests = [
        ("testYellowBananaIsRipe", testYellowBananaIsRipe),
    ]

    func testYellowBananaIsRipe() {
        let banana = Banana(color: .yellow)
        XCTAssertTrue(banana.isRipe)
    }
}

LinuxMain.swift

The package manager creates another file named LinuxMain.swift that acts as the test runner on non-Apple platforms. It contains a call to XCTMain(_:) in which you must list all your test suites:

// Tests/LinuxMain.swift
import XCTest
@testable import BananaKitTests

XCTMain([
    testCase(BananaTests.allTests),
])

Manual maintenance is easy to forget

This approach is obviously not ideal because it requires manual maintenance in two places:

1. Every time you add a new test, you must also add it to its class’s allTests.
2. Every time you create a new test suite, you must add it to the XCTMain call in LinuxMain.swift.

Both of these steps are easy to forget. Even worse, when you inevitably forget one of them it’s not at all obvious that something is wrong — your tests will still pass on Linux, and unless you manually compare the number of executed tests on macOS vs. Linux you might not even notice that some tests didn’t run on Linux.

This happened to me numerous times, so I decided to do something about it.

Swift 4.1 can update allTests for you

> swift test --generate-linuxmain

Safeguarding Linux tests against omissions

Let’s try to build a mechanism that automatically causes the test suite to fail when we forget one of the maintenance steps. We’re going to add the following test to each of our XCTestCase classes (and to their allTests arrays):

class BananaTests: XCTestCase {
    static var allTests = [
        ("testLinuxTestSuiteIncludesAllTests",
         testLinuxTestSuiteIncludesAllTests),
        // Your other tests here...
    ]

    func testLinuxTestSuiteIncludesAllTests() {
        #if os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
            let thisClass = type(of: self)
            let linuxCount = thisClass.allTests.count
            #if swift(>=4.0)
                let darwinCount = thisClass
                    .defaultTestSuite.testCaseCount
            #else
                let darwinCount = Int(thisClass
                    .defaultTestSuite().testCaseCount)
            #endif
            XCTAssertEqual(linuxCount, darwinCount,
                "\(darwinCount - linuxCount) tests are missing from allTests")
        #endif
    }

    // Your other tests here...
}

This test compares the number of items in the allTests array to the number of tests discovered by the Objective-C runtime and will fail if it finds a discrepancy between the two, which is exactly what we want.

(The dependency on the Obj-C runtime means the test only works on Apple platforms — it won’t even compile on Linux, which is why we need to wrap it in the #if os(macOS) ... block.¹)

A failing test when you forget to add a test to allTests

To test this out, let’s add another test, this time “forgetting” to update allTests:

import XCTest
import BananaKit

class BananaTests: XCTestCase {
    static var allTests = [
        ("testLinuxTestSuiteIncludesAllTests",
         testLinuxTestSuiteIncludesAllTests),
        ("testYellowBananaIsRipe", testYellowBananaIsRipe),
        // testGreenBananaIsNotRipe is missing!
    ]

    // ...

    func testGreenBananaIsNotRipe() {
        let banana = Banana(color: .green)
        XCTAssertFalse(banana.isRipe)
    }
}

When we now run the tests on macOS, our safeguard test will fail:

I really like this. Obviously, it only works if you want the allTests array to contain every test, i.e. you’ll have to wrap any Darwin- or Linux-specific tests in conditional compilation blocks as we did above. I believe this is an acceptable limitation for most code bases.

Safeguarding LinuxMain.swift

What about the other problem, verifying that LinuxMain.swift is complete? This is harder. LinuxMain.swift is not (and cannot be) part of the actual test target, so you can’t easily verify what gets passed into XCTMain.

The only solution I can see would be to add a Run Script build phase to your test target and have the script parse the code in LinuxMain.swift and somehow compare the number of items in the array to the number of test suites in the test target. I haven’t tried it, but it sounds quite complicated.

Conclusion

Even with the new test, things are far from perfect since there are still two things you can potentially forget. Every time you create a new XCTestCase class, you must:

1. Copy and paste the testLinuxTestSuiteIncludesAllTests test into the new class.
2. Update LinuxMain.swift.

Still, I think this is considerably better than the status quo because the new test covers the most common case — adding a single test to an existing test suite and forgetting to update the allTests array.

I can’t wait for Swift’s reflection capabilities to become more powerful², making all this unnecessary.