Why do my builds take 3 times as long on Windows?

Poor performance, small files, and deleting things (slowly)

At a previous internship, I worked on a web app with a semi-complicated build process.

I worked on a part of the frontend that was built with a combination of Maven, NPM, and Grunt. Whenever I wanted to test stylesheet or JavaScript changes, I usually had to run the entire build process with mvn clean install. Since it used multiple build systems and non-thread-safe Maven plugins, I couldn’t take advantage of incremental builds or Maven’s parallel builds feature.

That meant my edit-compile-run cycle was pretty long. For example, if I changed a single line of Less or JavaScript and then ran mvn clean install, I could expect it to take between five and six minutes. As you might imagine, waiting again and again to test small changes could be frustrating. To work around this, my team had gotten in the habit of iterating in the browser developer tools before making changes to the actual source, but that was suboptimal too—the CSS that made it to the browser was missing Less variables and the JavaScript was organized differently.

I didn’t want to completely tear apart the build process to fix the performance issues, especially since I was only at the company for a summer internship (and I had a feeling that “make builds faster” didn’t crack my manager’s top ten priorities). However, I figured there had to be at least one major bottleneck that I could eliminate without too much hassle.

My work laptop ran Windows 11. I vaguely remembered that Windows is often slow in two particular cases:

• Operations on lots of small files, rather than on fewer large files. This is especially noticeable when you’re copying/moving/deleting files in Explorer: the progress bar slows down dramatically when it runs into a patch of many small files and then speeds back up as soon as it gets to a big one. Most operating systems and file systems are faster when dealing with a few big files than many little ones, but the contrast seems to be especially pronounced on Windows.
• Starting lots of new, short-lived processes. Running Bash scripts in Git Bash (or even waiting for the shell prompt to load) is often painfully slow. I would guess that part of this has to do with emulating the fork() system call, since Windows lacks a direct equivalent, but perhaps also that creating processes is more expensive to begin with.

Builds involve tons of small file operations and lots of temporary subprocesses, so I had a hunch that my workload brought out the worst-case performance characteristics in both areas.

To test my theory, I decided to try Windows Subsystem for Linux for comparison. Today there are two variants of WSL with different characteristics. WSL 1 translates Linux system calls to Windows ones, while WSL 2 runs a real Linux kernel in a Hyper-V virtual machine. According to the documentation, WSL 2 offers much better file system performance, so I went with that.

Comparing WSL 2 performance to native Windows

To get the biggest benefit from using WSL 2, I cloned the Git repository again in my Linux home directory. I was pleased to see that the build worked on Linux without modifications and it felt much faster.

I wanted to be a little more scientific than that, so I ran mvn clean install five times under each environment. I ran all the Windows builds and then all the Linux builds rather than switching back and forth, hoping that each operating system might do some caching to make subsequent runs faster.

Here’s what I got:

So on average, building the same project on Windows takes over 3.5 times longer! Curiously, the Windows results are also a lot less consistent. The Linux results are about 13% faster after the first run (with warm caches), but you can’t see the same effect on the Windows side.

While watching the lines of output scroll by, one thing stood out to me: deleting the node_modules folder on Windows as part of the “clean” phase took much longer than doing the same inside WSL 2. (I don’t know why the build process had to delete and reinstall all the NPM dependencies every time, but someone else had previously tried and failed to fix that, so I didn’t investigate further.) Although this delete operation certainly wasn’t enough to singlehandedly explain the slowness on Windows, it seemed worthy of investigation on its own.

Zooming in on the delete operation

The answers to this Super User question say that deleting files on Windows can take more or less time depending on how you do it. Piecing together the various suggestions, it seems that the Delete button in Explorer is slowest, the del command is faster, and robocopy’ing an empty directory is fastest.

This disparity suggests that there are varying levels of overhead involved. Time for another experiment!

This time, I’m using the same node_modules folder with 62,528 total items, generated fresh each time with npm i. I timed the Windows commands with some PowerShell trickery since Windows doesn’t have time like Unix does. For the Explorer measurements, I just used a stopwatch as best I could.

# rd
Measure-Command {start-process cmd -argumentlist "/c rd /s /q node_modules" -wait}

# robocopy
mkdir empty
Measure-Command {start-process cmd -argumentlist "/c robocopy empty node_modules /MIR /R:1 /W:1 /MT:128 /LOG:robocopy.log" -wait}
		

As expected, Explorer was the slowest. Perhaps surprisingly, rd is actually faster than robocopy. This comparison once again produced more questions than answers: what could Explorer be doing that makes deleting files take twice as long as other tools? And more importantly, why does it take 20 to 40 times as long to delete files on Windows than on Linux?

I suspected that the answers might involve filter drivers, the lack of a directory entry cache, and fundamental architectural decisions in NT. Unfortunately, by this point, I had to get back to my usual work, so I didn’t continue looking for answers to my questions. I was happy that I found a way to make my builds over three times faster. The time I had spent switching to WSL paid off many times over in the next few weeks.

As for Maven’s clean operation in particular, I later discovered that Maven does not, in fact, shell out to any of these command-line tools. Instead it has its own deletion code which includes such features as “fast delete” (which copies an empty directory) and some sort of exponential backoff for deleting files that Windows thinks are in use.

If I get more time to work on similar issues in the future (perhaps when it affects both my company’s revenue and developer productivity), I’d love to do some digging with the performance analysis tools in Windows. To Microsoft’s credit, it sounds like there are great tools for Windows profiling, and the debugging symbols are freely available. Bruce Dawson’s profiler antics have been an inspiration; this is a cool area I didn’t learn about in my CS degree and I’d love to dive in at some point.

For now at least, I’ll use this as a reminder that curiosity—just asking “why does it have to take so long”—often pays dividends.