From time to time there are discussions on LinuxQuestions or IRC about making SlackBuilds run faster using “multiple jobs” that will use all the cpus in your system.
The good news is that it usually works. You can tell GNU make to use “multiple jobs” in parallel by setting an environment variable (see below). Most upstream software is built using GNU make, including all the builds that take a very long time, so this solution works well.
The bad news is that, for some upstream software, it sometimes fails.
How do I build with multiple jobs?
For SlackBuilds from SlackBuilds.org, this will work in most circumstances (the nproc command returns the number of cpus in your system).
Some of the build tools, and other SlackBuilds repositories, may have other options.
- For sbopkg you can set MAKEFLAGS in sbopkg.conf
- For sbotools you can set ‘-j’ with sboconfig
- For many Slackware SlackBuilds you can set the NUMJOBS environment variable
- For many slacky.eu SlackBuilds you can set the JOBS environment variable
Note that there will be some stages of the build where not all of your cpus will be used, for example “configure” at the beginning, and linking and package creation at the end.
Don’t use too many jobs. It’s not really a problem to have many more jobs than cpus, but each additional job will need more and more memory. Eventually you will run out of memory and the build will fail.
GNU make also has a new alternative to
For example, if you specify
MAKEFLAGS="-l2.5", GNU make will only start more jobs
if the system’s load average is less than 2.5.
Why does it fail sometimes?
Failures happen when output from one job is used as input by another job that doesn’t wait properly. If the first job hasn’t finished, the second job will fail. Running with only one job prevents the problem.
Sometimes a build with multiple jobs will fail on one system, even if it is ok on another system. It depends on the characteristics of the system, including the number of cpus, hyperthreading, cache sizes, storage speed, etc.
Success or failure sometimes changes, as bugs in upstream Makefiles and in GNU make get fixed and new bugs appear.
Some upstream software simply hasn’t been written to use multiple make jobs.
What errors will I see when a build fails with multiple jobs?
- Errors about missing files or directories
- Errors linking libraries that don’t exist
- Builds that go very slow and then run out of memory
Writing SlackBuilds for multiple jobs
If you are writing a SlackBuild for SlackBuilds.org, please do not do anything.
- If you don’t do anything, the user will be able to use the MAKEFLAGS environment variable, and the options in sbopkg, sbotools, etc.
- But if your SlackBuild fails with multiple jobs, please use a hardcoded
make -j1command. You might also need
make -j1 install.
- Sometimes you will see
make || make -j1(that is, if make fails, try again with ‘-j1’). This might confuse inexperienced users, and might annoy people with slow systems, so please don’t use this code unless you specifically need it.
- Please don’t invent some new environment variable instead of MAKEFLAGS.
- Please don’t copy the code from Slackware’s SlackBuilds. The default is good for Patrick’s own professional build system and workflow (he uses ‘-j7’), but it’s not good for people with old or small systems.
- For the same reason, please don’t hardcode something like ‘-j3’. It might be good for you, but it’s not good for everybody.
- Please don’t calculate the “right” number of jobs from the number of CPUs. The user might want to do something else while the build is running in the background, or might be worried about overheating, or might want to use ‘-l’ instead of ‘-j’.
Which SlackBuilds fail with multiple jobs?
SlackBuilds from SlackBuilds.org that almost always fail with multiple jobs usually have ‘-j1’ hardcoded in them, so they will build even if you have set the MAKEFLAGS environment variable.
But because everybody’s system is different, and because bugs come and go, it is very difficult for the SBo maintainers and admins to set ‘-j1’ in all the SlackBuilds that need it.
Here’s a list of SlackBuilds that sometimes seem to fail with multiple jobs for me. This list is for building on Slackware 14.2.
academic/abntex academic/arpack academic/gretl academic/staden academic/tophat academic/units audio/Pd-extended audio/dssi-vst audio/flake audio/non desktop/ClipboardViewer desktop/afterstep desktop/avant-window-navigator desktop/ee desktop/ion desktop/lumina desktop/menumaker development/bmake development/camlp5 development/chibi-scheme development/chicken development/ecl development/gauche development/gengetopt development/gforth development/gnats development/google-go-lang development/gprolog development/icon development/latrace development/menhir development/ming development/nvi development/polyml development/regina-rexx development/rmlmmc development/simulavr development/smlnj development/sourcenav development/splint games/adl games/armagetronad games/dopewars games/ember games/lmpc games/megaglest games/openmsx-catapult games/pinball games/tome4 gis/saga-gis graphics/tuxpaint libraries/agg libraries/libb64 libraries/librfm5 libraries/stfl libraries/xmlrpc-c libraries/xview multimedia/dvdrip multimedia/lives network/dsniff network/gtk-gnutella network/guacamole-client network/milter-greylist network/openl2tp network/sslh network/thrift network/tightvnc office/color-theme office/minidjvu office/wyrd system/ZoneMinder system/collectd system/dieharder system/dvdisaster system/dynamips system/efitools system/freeswitch system/heirloom-doctools system/hercules system/hwinfo system/icinga-web system/mdocml system/munin-node system/opam system/passwordsafe system/rhash system/syslog-ng system/uae system/unison system/xen