SCCM Gotcha: Task Sequence Media Size and FAT32

In SCCM, you have the option to create Task Sequence Media, which allows you to build a flash drive, DVD, or other bootable media that you can use to deploy operating system task sequences when no network connection is available (specifically, this is "stand-alone media" mode, versus "bootable media" and other options). This is very useful for off-site situations where you may not have any access to your organization's central network, or for computers whose NICs are not working correctly, but still need to be imaged.

In our environment, we tend to use USB flash drives for Task Sequence Media, and I was recently asked to create one for the first time with my newly-granted SCCM Administrator permissions. So, I went through the Create Task Sequence Media wizard, following the steps in the Microsoft docs, but it repeatedly failed, complaining that the USB media did not have enough space available to hold the task sequence. But that didn't make sense - I was using a 64 GB flash drive, and yet when I looked at it in the Disk Management console, I saw that it was only formatted with a single 32 GB partition.

It turns out that the reason for this is that Microsoft does not support partition sizes of greater than 32 GB on FAT32-formatted drives on anything other than Windows NT 3.51. As Wikipedia notes, FAT32 can actually support up to 8 TB or 16 TB partitions, so it's not totally clear why Microsoft chooses to impose this limit. Some conspiratorialists online claim that this was a move to force users onto Microsoft's then-new NTFS filesystem; others say that it has to do with USB flash drives' internal specifications. Whatever the reason, the fact remains that the SCCM Task Sequence Media wizard only offers to format a USB flash drive as FAT32.

This means that if you have a task sequence whose contents are near to or greater than 32 GB, you are likely to encounter the same problems I did. In our case, it is because of the vast range of hardware that our imaging drives must support, causing the driver packages to take up almost 20 GB, but in situations with a lot of software - CAD, Adobe Creative Cloud, etc. -  you could hit the limit with just one driver package.

The solution seems to be not to ask the wizard to create a bootable USB flash drive for you. Instead, choose the second option, "CD/DVD set." The default media size is 4.7GB, but you can increase this, all the way up to "unlimited." Then, using the Browse button, you select the location to which the wizard should save the resulting .ISO file, and continue through the wizard like normal. Once the .ISO is built, you can use a tool like Rufus to copy it to a USB flash drive of an appropriate size and make it bootable. Once you've confirmed that it's working, you can use another tool like ImageUSB to capture a binary copy of that flash drive, and then write it concurrently to a batch of drives.

Don't use an ampersand in the title of a macOS installer package

I recently created a macOS installer for an application using Jamf Pro's Composer, and was disappointed to find that when I went to run it, I got an error reading:

The operation couldn't be completed.
(com.apple.installer.pagecontroller error -1.)

I examined the name of the package I had created. This application has an ampersand in its name, and I put its version number, 6.0, at the end of the package name. I know that the period in the version number does not cause any issues, nor do spaces in the name, so the only odd one out is the ampersand. I removed it, rebuilt the package, and found that I could install it without issue.

So, in summary, don't use an ampersand in the title of a macOS installer package, or you'll get a very unhelpful error.

Barco ClickShare and Microsoft Edge on Windows 10 build 1803+

If you have a computer that upgrades from Windows 10 build 1709 to build 1803 or newer, and Edge stops working, check to see if you have the Barco ClickShare extension pack installed. If you do, uninstall it, and install the updated version 1.1, which Barco claims should resolve this issue.

Using NoMAD to Replace Traditional AD Binding on macOS

Another day, another problem with Apple's implementation of Active Directory bindings on macOS. Our current issue is that employee laptops, which are bound to AD, do not reliably authenticate over WiFi. We have applied a Configuration Profile that is supposed to use the credentials entered on the login screen to sign in to WiFi, which should then allow it to authenticate to AD, but this does not work consistently, either - only some accounts can log in, and even then only after a few tries. Our networking team has confirmed all of our settings, so it seems like there is some fundamental issue with the combination of macOS, AD bindings, and our WiFi setup.

One potential solution to this chicken-and-egg problem is to create a special account that Macs could use to pre-sign in to WiFi on the login screen, which would allow users to enter their AD credentials and log in just like if they were on Ethernet. You could then use a logon Policy to disconnect from the WiFi network, and then the user would log in with their own credentials. However, this doesn't address some of the other problems that AD-bound Macs have, such as the problems in synchronizing their account credentials with FileVault 2, and the difficulty with SecureTokens in newer versions of macOS if Macs are set up with DEP-based PreStage Enrollments, and therefore never have a local account manually created.

So, since Apple seems to be slowly breaking their AD implementation bit by bit with new features that don't integrate well with it, the more manageable solution is to forgo binding Macs to AD altogether - at least those that will be used primarily on WiFi, or those that need to have FileVault 2 encryption enabled. Local accounts just work better. Unfortunately, it's also a lot of work to make sure that users are updating their local account passwords when their Domain account credentials change, and eventually people may just give up.

NoMAD - from IT's perspective

Enter NoMAD. NoMAD is an application that replaces the traditional AD binding on a Mac with a combination of local accounts and a utility to help synchronize passwords between the two whenever either credential changes. In our implementation, we deploy the NoMAD installer .pkg using Jamf, along with a LaunchAgent to start the application whenever a user logs in, and a default Preferences file with settings provided by NoMAD for customization and setup purposes.

The LaunchAgent script is a simple cat command that creates a .plist file in /Library/LaunchAgents. In this location, the .plist is run for any user who logs into the machine, running under their own session and with their own credentials. Therefore, any changes that may be made in the application are applied only to that user, which is important for an application like NoMAD that deals with AD credentials.

The LaunchAgent script is similar to the following:

#!/bin/bash
cat > /Library/LaunchAgents/com.example.NoMAD.plist <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.NoMAD</string>
<key>LimitLoadToSessionType</key>
<array>
<string>Aqua</string>
</array>
<key>ProgramArguments</key>
<array>
<string>/Applications/NoMAD.app/Contents/MacOS/NoMAD</string>
</array>
<key>KeepAlive</key>
<true/>
</dict>
</plist>
EOF

NoMAD also supplies a .pkg that creates a LaunchAgent for you, if you prefer to use that instead.

We use another script to create a default preferences file, /Library/Preferences/com.trusourcelabs.NoMAD.plist:

#!/bin/bash
cat > /Library/Preferences/com.trusourcelabs.NoMAD.plist <<EOF
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>ADDomain</key>
<string>domain.com</string>
<key>KerberosRealm</Key>
<string>DOMAIN.COM</string>
<key>LocalPasswordSync</key>
<true/>
<key>UseKeychain</key>
<true/>
</dict>
</plist>
EOF

So, we combine the NoMAD installer .pkg with the two scripts above into a Policy, then add our laptops into the scope, and at the same time we unbind them from AD to prevent any possible issues. Deployment proceeds as with any other application.

One implication of this setup is that user accounts must be created locally before they can be used, in contrast to AD/mobile accounts that are created on first sign-on. This could be done via SSH or Jamf Remote, or manually, depending on the situation. Happily, since NoMAD handles synchronization of passwords between local accounts and AD, the technician who creates the account can just give it a password like "password", and NoMAD will prompt the user to change it after their first login.

NoMAD - from the user's perspective

The user will probably not interact directly with NoMAD very much - mostly, they only need to use it when their password changes, in order to update their local account to match. On first login, the user enters the local account credentials provided by IT in order to sign in to macOS. Once there, they sign in again, this time into NoMAD, using the menubar icon.

NoMAD prompts for the user's AD credentials, which it then uses to authenticate against the AD server specified in the Preferences file.

Once this is done, the Mac works more or less the same as if it were joined to Active Directory, with the added benefits that things like FileVault 2 password synchronization actually works as it should, since it is actually using local account credentials.

Fix "Network accounts are unavailable" on macOS Login Screen with a caffeinate LaunchDaemon

On Macs that are bound to Active Directory, you may from time to time see this popup next to the username box on the macOS login screen:

In most cases, assuming that the Mac is in fact connected to a wired Ethernet connection, this is not actually true, and if you try to log in with your AD credentials you will be successful, and if you wait 30 seconds or so, the message will go away. This message usually appears when the Mac has not been in use for a while and fallen asleep, or after booting up.

It seems that this is probably caused by a decision to show the login screen as quickly as possible, to the point that the network stack and/or AD connection is not fully initialized by the time it is visible. When you try to log in, it kick-starts the OS into loading everything, and should let you right in. However, if you are bothered by this, or have users who are, there is a simple fix that has some other benefits: creating a LaunchDaemon running caffeinate.

caffeinate is a macOS Terminal command that creates an "assertion" to keep certain parts of the computer from going to sleep. These assertions inform the System Management Controller (SMC) that the Mac should not sleep because an application requires it to stay awake, because a user is currently active, etc. Different assertions have different effects, which are reflected in caffeinate's different options. For example, to keep a Mac from putting its display to sleep, you can use caffeinate -d, and, to keep the disk from sleeping, use caffeinate -m.

To keep the Mac from going into idle sleep, which is the state in which the network stack appears to turn off, causing the popup, you can use caffeinate -i for desktop Macs, and caffeinate -s for mobile Macs. This difference is important: -i creates an assertion that does not let the Mac go into idle sleep at all; -s creates an assertion that is only valid when the system is running on AC power, meaning that a laptop will not stay awake in a bag, or other potentially damaging situation, as long as it is disconnected from its charger.

You can also use the caffeinate command to create an assertion that is only valid for a certain number of seconds, with caffeinate -t <seconds> -<mode>, or to create one that stays valid as long as a certain command is running, such as caffeinate -i make to keep the Mac awake as long as make is running.

So, with all of that out of the way, how can you use caffeinate to prevent that message from popping up on the login screen? Use a LaunchDaemon. LaunchDaemons run at system startup, and continue running regardless of whether a user is on the machine or not. I use a LaunchDaemon like this, located at /Library/LaunchDaemons, to keep the popup from appearing on our lab desktop Macs:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE plist PUBLIC "-//Apple Computer/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<dict>
<key>Label</key>
<string>com.example.caffeinate.plist</string>
<key>Program</key>
<string>/usr/bin/caffeinate</string>
<key>ProgramArguments</key>
<array>
    <string>/usr/bin/caffeinate</string>
    <string>-s</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>

Install that LaunchDaemon and reboot the Mac to make it take effect. The popup will still appear for 30 seconds or so after a reboot, as the OS hasn't had a chance to initialize everything yet, but it should not reappear afterwards. This still allows the screen to go to sleep, as well as the HDDs, but keeps the rest of the OS awake so that the popup does not appear.

Since this keeps the networking components of the OS awake, it also helps resolve situations where pinging a Mac that has gone to sleep can take several tries as it tries to wake up. This, therefore, makes the process of getting Macs to check in to Jamf or another management system more reliable.

Data recovery with Recuva

When a file is deleted in most file systems, the portion of the disk that holds the file is marked as "free", and is available for other files. However, the actual data - the ones and zeroes that make up the file - remain until they are overwritten. This means that if you accidentally delete a file, and even if (on Windows) you empty the Recycle Bin, there may still be hope to recover it.

One application that can allow you to recover deleted files is Recuva. Recuva works with Microsoft file systems (FAT12/16/32, exFAT, and NTFS), as well as Ext2, 3, and 4. This application provides a user-friendly, GUI-based wizard that makes it easy to choose what types of files you're trying to recover, and from where they are missing.

When you launch Recuva, you are welcomed to the wizard. Click "Next."

Next, the wizard prompts for the type of file or files you're trying to recover. If the file type you're looking for is not listed, you can pick the top option, "All Files." Choose your option, then click "Next."

Then, the wizard asks where the missing files were located, so that it will know where to scan. If you're not sure, or if they were in a location not listed, pick "I'm not sure", and Recuva will scan all storage devices on the computer. Choose the location, then click "Next."

Finally, the wizard gives you the option to enable "Deep Scan". By default, Recuva will search the portion of the disk that stores records that keep track of files - on FAT file systems, the FAT itself; on NTFS, the Master File Table; on Ext file systems, the inode table - in hopes of finding records that are marked "free". However, it is possible for records of deleted files to be missing from these file tables. In this case, Deep Scan causes Recuva to scour the entire drive for deleted files. While this is more thorough, and more likely to recover files, it will also take much longer than a regular scan. Check the box for Deep Scan if you choose, then click "Start."

Recuva then begins to scan all of the drives selected for the types of files you said were missing. Depending on the number of locations you chose, and the size of the drives to be scanned, this may take several minutes even with a normal, non-Deep scan.

Once scanning is complete, Recuva will show you a list of all of the deleted files it was able to find. The colored circle to the left of each filename indicates the likelihood of recovering each file: green means that the file has an excellent chance of full recovery; orange means an "acceptable" chance of recovery; and red means that it is unlikely that the file can be recovered. Click the checkboxes to the left of each file you wish to recover, then click "Recover..." in the bottom right to begin restoring these files.

If you would like more control or information about the files Recuva located, you can click "Switch to advanced mode" in the top-right of the window. In Advanced Mode, you can sort and filter results by name, path, and disk, as well as see a preview of the file and header information to help identify it. Like with normal mode, check the boxes next to the files you wish to recover, then click "Recover..." in the bottom right to begin restoration.

Recuva will prompt you for a location to save recovered files. Since deleted files are stored on parts of the disk that may be overwritten when new files are created, it would be a good idea to save the recovered files to a flash drive, external hard drive, or some other location than the disk from which you are recovering them.

Once the recovery is complete, Recuva gives a summary of how many files were fully or partially recovered. Review this, then click "OK."

You can verify the status of the recovered files, for most file types, by opening a File Explorer window and turning on the Preview Pane (in the View tab of the ribbon). Navigate to the directory where you saved the recovered files, and click on each one to bring it into the Preview Pane. If the file can be displayed, Explorer will show its contents; if it is too damaged, or of a type that can't be displayed, Explorer will say so.

Even if a file cannot be displayed correctly or opened in the appropriate application, you may still be able to recover data from it by opening it with Notepad or another text editor (Notepad++ is always a good choice.) This works better with some file formats than others: old Word documents (.doc), for example, often have large stretches of plain text, which may be readable with Notepad, while newer Word documents (.docx) are formatted in a way that makes this less likely.

Run commands on next startup using launchd and Jamf

As part of my solution for replacing traditional imaging on our Macs, I found the need to have a Jamf Policy run some commands the next time a Mac starts up. One way to do this is to have a separate Policy triggered on startup, with a script payload containing the commands you wish to run.

While this works great, it does create a situation where two sets of Jamf logs may need to be flushed should you need to re-run a Policy, and could lead to confusion if the same set of Macs aren't added to the scope of each Policy. You could solve that problem with a Smart or Static Computer Group, but that's another thing to keep track of, and another possible point of failure.

So, instead, I read up on launchd, Apple's replacement for the older Unix-style init system. launchd manages two different types of processes: LaunchAgents, which are run when a user logs into the computer, and LaunchDaemons, which are run when the computer starts up. Both LaunchAgents and LaunchDaemons can be configured to run either as a specific user, or as the root user. Both are configured using XML-based plist files with a variety of keys to specify programs to run, with what arguments they should be run, whether they should be kept alive after being loaded, and even conditional operators to control execution directly from the plist based on certain environmental triggers.

I already had a Policy set up that will remove all user and application data from the Mac (simulating a clean image), copy the latest macOS installer to the /Applications folder, and reinstall macOS. I wanted, upon reboot, for the Mac to automatically set its time from our NTP server, join itself to our Active Directory (placing itself into the correct OU based on its name), reset the Jamf management account's password, and check for Policies from Jamf so that it would reinstall any provisioned applications, etc. - all without requiring a tech's intervention after the Policy had kicked off.

Therefore, rather than adding complication in the form of additional Policies and Computer Groups, I decided to create a script that would build a LaunchDaemon for me directly from the reimaing Policy. I found on my first attempt that LaunchDaemons may run before the operating system's networking facilities are available, so commands needing network access - such as checking in with Jamf - would not work. Luckily, Jeff Kelley explains on his blog how to wait until networking is available using an Apple-provided function.

Here is the script I came up with to run commands the next time the system starts up:

#!/bin/bash
networkup='${NETWORKUP}' #have to insert this as a string into the script below, rather than letting bash try to give it a value, and return "" in the while below

# First, create the script that will run each of the specified commands at the next startup.
cat > /launchscript.sh <<EOF
#!/bin/bash

# Delay until networking is available.
#https://blog.slaunchaman.com/2010/07/01/how-to-run-a-launchdaemon-that-requires-networking/
. /etc/rc.common
CheckForNetwork

while [ "$networkup" != "-YES-" ]
do
        sleep 5
        NETWORKUP=
        CheckForNetwork
done

$4
$5
$6
$7
$8
$9
$(10)
$(11)

# Clean up this script, and the LaunchDaemon plist after the commands have been run.
rm /Library/LaunchDaemons/com.example.launchscript.plist
rm /launchscript.sh
EOF

# Next, make sure that the script is executable.
chmod +x /launchscript.sh

# Then, create a LaunchDaemon plist and register it so that the script will be run on the next startup.
cat > /Library/LaunchDaemons/com.example.launchscript.plist <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.example.launchscript</string>
    <key>Program</key>
    <string>/launchscript.sh</string>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardOutPath</key>
<string>/var/log/launchscript.log</string>
    <key>StandardErrorPath</key>
<string>/var/log/launchscript.log</string>
  </dict>
</plist>
EOF

This script takes up to eight command strings in variables $4 - $11, as is normal for Jamf scripts, and inserts them into another script, /launchscript.sh. That script waits until networking is available, runs the commands specified, then cleans up both itself and the LaunchDaemon that runs it on startup. Then, after making the launch script executable, the script creates a simple LaunchDaemon plist file in /Library/LaunchDaemons, which will be loaded and run on the next system startup. This LaunchDaemon redirects its standard out and error streams to a log file, /var/log/launchscript.log, so that any error messages that happen during launch script execution can be captured for debugging purposes.

This script accepts up to eight commands, but those can be used to run much larger scripts, or even to call Jamf policies or installer packages. I have not yet tried to run applications with this script, but I would guess that anything requiring a desktop environment would not run correctly. Since the LaunchDaemon and its associated launch script run as the system root user, you can use this to perform any sort of management or maintenance you would like on the computer, even without an administrator logging in.