Cookutils Documentation

de en fr pt ru

SliTaz Cook & Cooker

The SliTaz Cookutils provide tools and utils to help build SliTaz packages. They are easy to use and learn, fast and light. You will be able to create SliTaz packages in a few commands. The cookutils suite is split BSD-style, one tool for one task, around the central cook command and the Cooker.

cook compiles a receipt and produces a tazpkg. Around it, a handful of focused sub-tools handle the wok queries, the packages database, the build environment setup, batch building, dependency analysis, diagnosis, cleaning and receipt creation. The Cooker is a build bot with more automation and can be used as a frontend to cook since it provides a CGI/web interface to view logs in a colored way. All tools share the same DB files and wok, and they all share blocked and broken package lists as well as activity.

For technical information (coding style, repo layout, etc), refer to the README found in the source tree or in /usr/share/doc/cookutils. The receipt format is described in Receipts v2 and the receipt build options in COOKOPTS.

Tools overview

The cookutils suite is composed of the following user-facing tools, all available under /usr/bin/: 

cook              Compile and package a single receipt
cook-all          Batch-cook packages from a list / cookorder
cook-clean        Wipe wok artefacts (taz/, install/, source/)
cook-deps         Analyze runtime dependencies of a cooked package
cook-doctor       Diagnose the build environment or a package (read-only)
cook-new          Create a new package skeleton in the wok
cook-pkgdb        Build the package DB for $PKGS plus split.db / maint.db
cook-setup        Bootstrap a build chroot or cross-compile environment
cook-tui          Ncurses front-end: type a package name, watch it build
cook-wok          Read-only queries on the wok (list / search / uncook / …)
cooker            Build bot with CGI web interface
cookiso           ISO image builder
cooklinux         Kernel build wrapper
cooks             Cook a package, then its split packages that have their
                  own wok directory (v1 receipts)
cross             Cross-compilation toolchain builder (make install-cross)

The internal modules run by cook itself (precheck, postcheck, compressor, fix-desktop-file) live in /usr/libexec/cookutils/ and are not meant to be called by hand.

Every cook-* tool understands help, --help, -h or usage for its built-in usage screen with examples. cook and cooker take usage, help or -h (a cook --help is read as a package name).

For backwards compatibility, cook keeps transparent shims for the legacy sub-commands. They exec the matching cook-* tool, so existing scripts and habits keep working: 

cook setup [opts]      -> cook-setup [opts]
cook <arch>-setup      -> cook-setup <arch>
cook new <pkg> [-x]    -> cook-new <pkg> [-x]
cook doctor [...]      -> cook-doctor [...]
cook list <file>       -> cook-all <file>
cook all [opts]        -> cook-all [opts]
cook clean-wok         -> cook-clean wok
cook clean-src         -> cook-clean src
cook list-wok          -> cook-wok list
cook search <query>    -> cook-wok search <query>
cook uncook            -> cook-wok uncook
cook wanted            -> cook-wok wanted
cook build_depends     -> cook-wok build_depends
cook build_loop        -> cook-wok build_loop
cook pkgdb [--flavors] -> cook-pkgdb [--flavors]
cook splitdb           -> cook-pkgdb --splitdb
cook maintdb           -> cook-pkgdb --maintdb
cook <pkg> --deps      -> cook-deps <pkg>

Cook usage

Cook provides a small built-in help that you can display with the command usage, and a page of examples with howto. It also has options to perform special tasks on a package before cooking it or afterwards. To get help and usage: 

# cook usage
# cook howto

The options of cook <pkg>: 

--clean       -c   Clean the package in the wok
--getsrc      -gs  Get the package source tarball
--block       -b   Block a package so cook will skip it
--unblock     -ub  Unblock a blocked package
--pack             Repack an already built package (install/ must exist)
--continue         Keep the source tree and patches already applied,
                   run compile_rules again
--debug            Display debugging messages
--deps, --cdeps    Check the dependencies of a cooked package (cook-deps)

The short forms (-c, -gs, -b, -ub) must come right after the package name.

Howto

The first thing you will have to do before building packages is setup your environment. The 2 recommended ways of working: cook directly on host or cook in chroot to protect your host. In the case you want to work in a chroot you can install and use Tazdev to create one and chroot into it: 

# tazdev gen-chroot && tazdev chroot

By default Tazdev creates a chroot in /home/slitaz/cooking/chroot but you can specify a custom path in the argument (--arch=x86_64 puts it in /home/slitaz/cooking/x86_64/chroot). The chroot location is not important, when you will be in the chroot you will use standard SliTaz paths such as /home/slitaz/wok for the wok directory or /home/slitaz/log for all the cook logs. As usual you can display tazdev help usage with tazdev usage.

When you use a chroot there are 2 special directories mounted with the bind option: src and packages. The sources for all packages are stored by default in /home/slitaz/src, this directory is mounted into the chroot so the utils can use them. This method lets you share sources between many chroots such as one for cooking and one for stable. The packages directory default location is: /home/slitaz/[version]/packages so they are not in the chroot and are safe in case the chroot is removed by error.

When you cook directly on the host, cook protects it with a jail: an overlayfs with the running system read-only below and a throw-away branch in /dev/shm on top. Build dependencies are installed in the branch and vanish with it, only the wok, the logs and the packages are written for real. The bound filesystems are listed in JAIL_MOUNTS (cook.conf). A receipt can ask for its branch on disk instead of RAM with JAIL_NOT_RAMFS, or for no jail at all with JAIL_NOT_SUPPORTED. Without overlayfs (inside a plain chroot, for example) cook silently builds without a jail.

Getting started

Cook uses the /etc/slitaz/cook.conf configuration file, if you want to use custom paths for SliTaz directories and files, you'll have to modify it (a cook.conf in the current directory is read after it and overrides it). Setup creates some directories and files to keep trace of activity and errors, all files are pure plain text files that you can open in a text editor. It also installs the SETUP_PKGS and gives the slitaz group write access to $SLITAZ. To prepare your environment: 

# cook-setup

The cook-setup command has a --wok option which lets you clone a SliTaz wok while setting up your cook environment. Even if you are not yet an official developer you can clone it and use existing packages as an example to create your own. To setup and clone the default cooking wok or the undigest wok (--stable and --tiny clone the other woks, --forced reinstalls the setup packages): 

# cook-setup --wok
# cook-setup --undigest

For cross-compilation environments, pass the target architecture. This installs CROSS_SETUP and rewrites ARCH, CROSS_TREE, CFLAGS and HOST_SYSTEM in /etc/slitaz/cook.conf: 

# cook-setup arm
# cook-setup armv6hf
# cook-setup armv7
# cook-setup x86_64

cook.conf

The main settings of /etc/slitaz/cook.conf. The package keeps a local cook.conf on upgrade, and code defaults handle a missing variable, so an upgraded cookutils works with an existing cook.conf without any manual change. 

SLITAZ="/home/slitaz"   Root of WOK, PKGS, SRC, CACHE, LOGS, FEEDS.
ARCH="i486"             Target arch: i486, x86_64 or arm*. Sets CFLAGS
                        and the package names (-x86_64, -any suffixes).
SETUP_PKGS="..."        Packages installed by cook-setup / cooker setup.
SETUP_MD5=""            md5 of "ls $INSTALLED": when set, cook refuses to
                        start on a base that drifted (bdeps left over).
QA="0"                  Any non-empty value (even "0") checks the receipt
                        before the cook (precheck); empty disables it.
KEEP_BUILD_TREE=""      "yes" keeps source/ and install/ after a
                        successful cook (local development).
LOCALE=""               Translations kept in packages (empty: the
                        supported SliTaz locales).
JAIL_MOUNTS="/ /proc …" Filesystems bound into the build jail (was
                        AUFS_MOUNTS before 4.13, the old name stops cook).
MAKEFLAGS, CFLAGS, CPPFLAGS, LDFLAGS, CONFIGURE_ARGS
                        Build flags exported to every receipt.
DEFAULT_LOG_LIMIT=50    Max log size in MB per receipt function.
HOST_WGET="wget"        wget used for Repology (needs HTTPS).
MAKE_BUNDLE="no"        cook-pkgdb builds bundle.tar.lzma (build host
                        only, bakes upstream mirrors + extra.list).
REPOLOGY_CHECK="no"     postcheck queries repology.org for "outdated"
                        badges. Requires HTTPS-capable wget.
DEPS_CHECK="no"         postcheck runs cook-deps on the packed files and
                        prints the DEPENDS the receipt seems to lack.
DEPS_MIRROR_LIST="yes"  cook-deps also reads the mirror files list; "no"
                        on a build host whose packages dir is the mirror.

The network toggles default to "no" so a fresh chroot doesn't try to reach the network during cook / cook-pkgdb. Build hosts flip them to "yes". Missing build dependencies are installed from the local packages, never cooked automatically.

Test your environment

Cook provides a test command which will create a package and cook it. This lets you see if your environment is working and it provides an example package with a receipt. The dummy package is named 'cooktest' and can be removed after testing. To cook the test package: 

# cook test

Create and cook

If your environment is setup correctly you can start creating and compiling SliTaz packages from your wok. To create a new package with an empty receipt (you can also create one interactively): 

# cook-new pkgname
# cook-new pkgname --interactive

If you have just created a new package, you'll have to edit the receipt with your favorite text editor: the template is a receipt v2. When the receipt is ready or if you have an existing package, you can cook it: 

# cook pkgname

If all went well you will find your package in the $SLITAZ/packages directory and the cook log in $SLITAZ/log/pkgname.log. After a successful cook the wok keeps only the receipt, stuff/ and taz/ (the packed tree, shown by the Cooker web interface): source/ and install/ are removed, unless KEEP_BUILD_TREE="yes". A failed cook keeps everything for debugging.

Receipts

A receipt is a shell script with variables (PACKAGE, VERSION, DEPENDS, BUILD_DEPENDS...) and functions: compile_rules() builds the sources into $install, genpkg_rules() copies the files of each package from $install to $fs. There are two formats: 

# SliTaz package receipt v2.

PACKAGE="attr"
SPLIT="attr-dev"

genpkg_rules() {
    case $PACKAGE in
        attr)     copy @std ;;
        attr-dev) copy @dev ;;
    esac
}

In v2, split packages do not inherit DEPENDS, TAGS, CONFIG_FILES... from the top of the receipt, and a plain post_install() goes into every package: suffix it with the package name. Read Receipts v2 for the details, the copy() patterns, compiling sets and the checklist to convert a v1 receipt.

Install a cooked package

There is no install option: cook produces packages, installing them is the job of the package manager. In a build chroot, cook updates the packages that are already installed there with the fresh build, so the build environment stays current. On a native host it never does: the running system boots on what was just cooked (kernel, glibc, boot scripts). Install the package yourself: 

# tazpkg install $SLITAZ/packages/pkgname-version*.tazpkg

Get sources, repack, debug

If you want or need to download only the source of a package without building it, you can use the option --getsrc as below. The cook checks the tarball against TARBALL_SHA256 (or _SHA1, _SHA3, _SHA512, _MD5) and, when the receipt has none, prints the line to paste in it: 

# cook pkgname --getsrc

To pack again an already built package after a genpkg_rules change, without compiling, or to run compile_rules again on the existing source tree (patches already applied are skipped): 

# cook pkgname --pack
# cook pkgname --continue

Clean packages

After compilation and packaging there are several files in the wok that take up disk space. To clean a single package: 

# cook pkgname --clean

You can also clean the full wok at once or just remove the sources to reclaim disk (src keeps the trees of broken packages for debugging): 

# cook-clean wok
# cook-clean src

Search and list the wok

cook-wok handles read-only queries on the wok. It uses grep and so supports regular expressions: 

# cook-wok list                  # list packages in the wok (ARCH-filtered)
# cook-wok search libssh         # grep package names
# cook-wok uncook                # full inventory of packages without a tazpkg
# cook-wok wanted gtk            # cooker tasks for gtk* receipts (incremental)
# cook-wok build_depends         # idem driven by BUILD_DEPENDS
# cook-wok build_loop            # receipts with a build dependency loop (slow)

Receipt functions

Many packages provide the same kind of files such as *-dev packages with static libs, pkgconfig files and include headers. cook provides functions to be used in a receipt. In genpkg_rules: 

copy PATTERN...           Copy files from $install to $fs: @std, @dev,
                          folder/, file, wildcards, @rm, @ico
                          (see receipts v2)
get_dev_files             Copy /usr/include, /usr/lib/pkgconfig and the
                          static libs (/usr/lib*/*a) to $fs
cook_copy_files NAME...   Copy the files matching NAME, keeping paths
cook_copy_folders NAME... Copy the folders matching NAME, keeping paths
cook_copy_icons [SIZE...] Copy hicolor icons (default: 16 48)
remove_already_packed     Remove from $fs files already packed by a
                          previous package of the same receipt (v2)

In compile_rules: 

fix ld                    Add -Wl,--as-needed to LDFLAGS
fix libtool               Same in the generated libtool (after configure)
fix math                  C++ std::isnan... build fix for recent glibc
fix symlinks              Make absolute symlinks in $install relative
fix utf-8                 Install the en_US.UTF-8 locale for the build
fix gem                   Tidy a Ruby gem install (docs, man pages)
cook_pick_manpages FILE.. Copy man pages to $install/usr/share/man/manN
cook_pick_docs FILE...    Copy docs to $install/usr/share/doc/$PACKAGE-$VERSION
cook_perl                 Build and install a Perl module (Makefile.PL)

Patches listed in stuff/patches/series are applied automatically before compile_rules (one per line, [options|]name-or-URL[|sha256=...]). The receipt paths are $src (sources), $install (also $DESTDIR), $fs (the package file system) and $stuff. COOKOPTS tunes the packing steps (strip, compressors, desktop files...), see cookopts.txt.

Packages DB list

cook-pkgdb generates the package database for the $PKGS directory and refreshes the wok-wide split.db and maint.db. This lets you create a local packages repository quite easily and is used to create the official SliTaz packages list found on the mirrors. To create a packages list and the Live flavors files: 

# cook-pkgdb              # full rebuild
# cook-pkgdb /path/pkgs   # full rebuild against another packages dir
# cook-pkgdb --flavors    # full rebuild + regenerate TazLiTo flavors
# cook-pkgdb --rmpkg      # full rebuild + remove stale tazpkg files
# cook-pkgdb --splitdb    # refresh just $cache/split.db
# cook-pkgdb --maintdb    # refresh just $cache/maint.db

When called via --flavors, cook-pkgdb checks for a flavors repo in /home/slitaz/flavors and packs all flavors in /home/slitaz/live using the latest packages list available.

Batch builds

cook-all cooks packages listed in a plain text file, one per line. Lines starting with # and blank lines are skipped. With no file argument, ./cookorder is tried first, then /etc/slitaz/cookorder: 

# cook-all                        # use the default cookorder
# cook-all my.list                # cook from a custom list
# cook-all --resume               # restart a broken full rebuild

The --resume flag skips packages that already have a taz/ dir, which lets you continue after an interrupted rebuild without recooking what's already done. Each result is logged in $SLITAZ/log/cook-all.log, and the exit status is non-zero when a package failed.

Runtime dependencies

cook-deps analyzes runtime dependencies of a cooked package by reading its files.list and resolving each binary, shared library, pkg-config or libtool reference back to the package that ships it: 

# cook-deps libssh                # pretty per-sub-package output
# cook-deps libssh -q             # one line per sub-package, machine-friendly
# cook-deps libssh --la           # also follow *.la dependency_libs
# cook-deps libssh --incl         # also show glibc-base / gcc-lib-base

Diagnosis

cook-doctor runs read-only health checks: config, cache DB files, disk space, orphan jails, toolchain and SETUP_MD5, arch coherence, permissions. For each problem it prints a suggested fix command, but never runs it. On a package it checks the receipt, the build dependencies, the broken state with a log tail, the cooked tazpkg and the source tarball. The exit status is 1 when a problem is found: 

# cook-doctor                     # the build environment
# cook-doctor busybox             # one wok package
# cook-doctor all lib             # scan $WOK/lib*, one line per problem

The Cooker

The Cooker is a Build Bot, its first function is to check for commits in a wok, create an ordered cooklist and cook all modified packages. It can also be used as a frontend to cook since they both use the same files. The Cooker can also be used to cook a big list of packages at once such as all the packages in a flavor. The Cooker provides a nice CGI/Web interface that works by default on any SliTaz system since it provides CGI support via the Busybox httpd web server.

The Cooker provides a small built-in help usage and short command switch. For example to display usage you can use: 

# cooker usage
# cooker -u

The Cooker commands (short switch first): 

-u  usage                  Display the usage
-s  setup                  Setup the Cooker environment
    setup-cron [hours]     Add the Cooker to root's crontab (default: 2)
    check-cron             Show the Cooker cron jobs
    arch-db                Create the host arch packages DB
-n  note "text"            Add a note to the cooknotes
-ns notes                  Display all the cooknotes
-b  block pkg              Block a package so cook will skip it
-ub unblock pkg            Unblock a blocked package
-R  reverse pkg            Cook all reverse dependencies of a package
-p  pkg pkg                Same as 'cook pkg' but with cooker log
-f  flavor name            Cook all packages of a flavor
-l  list file              Cook all packages in the given list
-c  cat category           Cook all packages of a category
-r  rev number             Cook packages of a specific Hg revision
-a  all                    Find and cook all unbuilt packages
-T  tasks                  List existing cooker tasks
-t  task name              Execute the specified task
-o  outgoing               Find changes in wok that we can move to wok-hg
    autodeps               Find dependencies for all packages in wok

Cooker setup

Like cook, the Cooker needs a working environment before starting to use it. The main difference with the cook environment is that the Cooker needs 2 woks. One Hg and clean wok as a reference ($SLITAZ/wok-hg) and one build wok ($SLITAZ/wok). In this way it is easy to compare both woks and get modifications. If you already have a cook environment, you must move your wok before setting up the Cooker or it will complain. Setup will also install a set of development packages that can be configured in the cook.conf configuration file and the variable SETUP_PKGS, plus mercurial, rsync and tazlito, and clone the flavors repo. To setup your cooker environment: 

# cooker setup

If all went well you now have 2 woks, base development packages installed and all needed files created. The default behavior is to check for commits, you can run a test: 

# cooker

Cooker cook

Again, 2 ways to work now: make changes in the clean Hg wok and launch the cooker without any arguments or cook packages manually. The cooker lets you cook a single package or all packages of a category or a flavor. You can also try to build all unbuilt packages, but be aware the Cooker was not designed to handle thousands of packages.

To cook a single package which is the same as cook pkgname but with more logs: 

# cooker pkg pkgname

To cook more than one package at once you have different kind of choices. You can use an existing package list such as used for Live flavors, you can also use a custom list using the package names listed line by line. You can build all packages of a category, all the packages that depend on a package, or all unbuilt packages. 

# cooker flavor [name]
# cooker list [/path/to/cooklist]
# cooker cat [category]
# cooker reverse [pkgname]
# cooker all

The Cooker lets you also recook a specific Hg revision. It's useful in production so that if the Build Bot was interrupted while cooking commits, you can then cook packages manually: 

# cooker rev 9496

Blocked packages

Cook and the Cooker handle a file with a list of blocked package so they will not cook when commits happen or if a cooklist is used. This is very useful for a Cooker Build Bot in production. When you block or unblock a package you can add a note to the cooknotes. Blocking packages example: 

# cook pkgname --block
# cooker block pkgname
# cooker -n "Blocked pkgname note"

The list of blocked packages are also displayed on the Cooker web interface. To unblock a package you have to use the unblock command or cook --unblock option: 

# cook pkgname --unblock
# cooker unblock pkgname

Cooker CGI/Web

To let you view log files in a nice way, keep trace of activity and help find errors, you can use the Cooker Web interface installed by default in /var/www/cgi-bin/cooker (with a /var/www/cooker link). If you don't use a chroot and the Busybox httpd web server is running, the web interface will work without configuration and should be reachable at: http://localhost/cooker/cooker.cgi

If you used a chroot environment, you should also install cookutils on your host and modify the SLITAZ path variable. A standard working way is to have a chroot in: 

/home/slitaz/cooking/chroot

With /etc/slitaz/cook.conf modified as below: 

SLITAZ="/home/slitaz/cooking/chroot/home/slitaz"

Note: It's not obligatory to install the cookutils on your host to use the web interface. If you use Lighttpd you can also copy the cooker.cgi and style.css files for example into your ~/Public directory and use a custom cook.conf with it. The advantage of installing cookutils on the host is to get regular updates via the Tazpkg packages manager. Say you have cloned or downloaded the cookutils: 

$ cp -a cookutils/web ~/Public/cgi-bin/cooker
$ cp -f cookutils/configs/cook.conf ~/Public/cgi-bin/cooker

Edit the configuration file: ~/Public/cgi-bin/cooker/cook.conf to set your SLITAZ path and you're all done!

Cooknotes

The cooknotes feature lets you write small personal notes about packaging and is useful for collaboration. The cooknotes was coded to let the SliTaz Cooker bot maintainers share notes between themselves and other contributors. The Cooker can block a package's build or recook packages manually, for example it's nice to make a note if a package is blocked so that the maintainer knows why admin did that. Cooknotes are displayed on the web interface and can be checked from a cmdline: 

# cooker note "Blocked pkgname due to heavy CPU load"
# cooker notes

Cooker as a Build Bot

The Cooker is designed to be a Build Bot for SliTaz, this means it monitors 2 woks, updates the Hg wok, gets the differences and cooks all packages that have been committed. The safer and cleaner way to run the Cooker as a Build Bot with cron is to use a chroot environment, but it can run directly on the host if you want.

To run The Cooker automatically you must use cron from the chroot. The setup-cron command adds the jobs to the root crontab (/var/spool/cron/crontabs/root) and restarts crond. Say you would like to run the Cooker every 2 hours: 

# cooker setup-cron 2
# cooker check-cron

It does not run the Cooker directly: every 2 hours cron touches $CACHE/cooker-request, and a job run every 5 minutes starts the Cooker when a request is pending. Another job cooks the packages listed in $CACHE/recook-packages, which the web interface fills.

Cooker BB started at boot

The Cooker environment and cron task can automatically be started at boot time. You must have the cookutils-daemon installed on the host and use a standard SliTaz installation to make it work properly (cooking goes in /home/slitaz/cooking). The daemon script will mount any virtual filesystems if needed as well as source and packages. Source files are in /home/slitaz/src and bound into the chroot so you can share package's sources between several versions (stable, cooking, undigest). If the package is not yet installed: 

# tazpkg get-install cookutils-daemon

To start the daemon you must have a cron file definition for root in the chroot, the daemon script works like all other system daemons and can be handled with: 

# /etc/init.d/cooker [start|stop|restart]