STXXL  1.4-dev
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Disk Configuration Files
Author
Timo Bingmann (2013-2014)

A main feature of the STXXL is to take advantage of parallel access to multiple disks. For this, you must define the disk configuration in a text file, using the syntax described below. If no file is found at the locations below, STXXL will by default create a 1000 MiB file in /var/tmp/stxxl on Unix or in the user's temp directory on Windows.

These are the locations STXXL will look for a disk configuration file on Linux/Unix systems, in order of precedence:

  • If the environment variable STXXLCFG specifies a file, this is used.
  • Then the current directory of the program is checked:
    • first for .stxxl.$HOSTNAME (for host specific configuration),
    • then for .stxxl (for general configuration).
  • Then the $HOME directory of the current user is checked (usual method):
    • first for $HOME/.stxxl.$HOSTNAME (for host specific configuration),
    • then for $HOME/.stxxl (for general configuration).
Warning
On many Linux distributions the $HOSTNAME variable is not exported. For the host specific configuration to work, you must add "export HOSTNAME" to your shell configuration (.bashrc).

On Windows systems, STXXL looks for a disk configuration file in the following directories:

  • If the environment variable STXXLCFG specifies a file, this is used.
  • Then the current directory of the program is checked:
    • first for .stxxl.%COMPUTERNAME%.txt (for host specific configuration),
    • then for .stxxl.txt (for general configuration).
  • Then the %APPDATA% directory of the current user is checked (usual method):
    • first for %APPDATA%/.stxxl.%COMPUTERNAME%.txt (for host specific configuration),
    • then for %APPDATA%/.stxxl.txt (for general configuration).
Note
In a default Windows 7 installation, %APPDATA% is C:\Users\<username>\AppData\Roaming
You can visit your %APPDATA% directory by simply entering "%APPDATA%" in the Windows Explorer address/location line.

Disk Configuration File Format

Each line of the configuration file describes a disk. Lines starting with '#' are comments.

A disk description uses the following format: 

disk=<path>,<capacity>,<fileio> <options>

Description of the parameters:

  • <path> : full disk filename.
    • In order to access disks STXXL uses file-based access methods (see below). Each disk is represented as a file
    • If you have a disk that is mounted in Unix to the path /mnt/disk0/, then the correct value for the full_disk_filename would be /mnt/disk0/some_file_name.
    • If the string contains "###" (three '#'), then these symbols are replaced by the current process id.
  • <capacity> : maximum capacity of the disk
    • the following size suffixes are recognized:
      • K, M, G, T, P (powers of 10),
      • Ki, Mi, Gi, Ti, Pi (powers of 2).
      • if a number has no suffix, M (megabyte) is assumed.
    • 0 means autogrow, and the file will be deleted afterwards.
  • <fileio> : STXXL has a number of different file access implementations, choose one of them (recommended ones in bold):
    • syscall : use read and write system calls which perform disk transfers directly on user memory pages without superfluous copying (currently the fastest method)
    • wincall : on Windows, use direct calls to the Windows API.
    • linuxaio : on Linux, use direct syscalls to the native Linux AIO interface.
      The Linux AIO interface has the advantage of keeping an asynchronous queue inside the kernel. Multiple I/O requests are submitted to the kernel at once, thus the kernel can sort then using its disk schedulers and also forward them to the actual disks as asynchronous operations using NCQ (native command queuing) or TCQ (tagged command queueing).
    • memory : keeps all data in RAM, for quicker testing
    • mmap : use mmap and munmap system calls
    • boostfd : access the file using a Boost file descriptor
    • fileperblock_syscall, fileperblock_mmap, fileperblock_boostfd : same as above, but take a single file per block, using full_disk_filename as file name prefix. Usually provide worse performance than the standard variants, but release freed blocks to the file system immediately.
    • simdisk : simulates timings of the IBM IC35L080AVVA07 disk, full_disk_filename must point to a file on a RAM disk partition with sufficient space
    • wbtl : library-based write-combining (good for writing small blocks onto SSDs), based on syscall
  • <options> : additional options for file access implementation. Not all are available for every fileio method. The option order is unimportant.
    • autogrow, noautogrow, autogrow=[off/on] : enables automatic growth of the file beyond the specified capacity, enabled by default except if raw_device.
    • direct, nodirect, direct=[off/try/on] : disable buffering in system cache by passing O_DIRECT or similar flag to open.
      This is recommended as it improves performance, however, not all filesystems support bypassing cache. With direct or direct=on, STXXL will fail without direct access. With nodirect or direct=off it is disabled. The default is direct=try , which first attempts to open with O_DIRECT and falls back to opening without if it fails.
    • unlink (or unlink_on_open) : unlink the file from the fs immediately after creation.
      This is possible on Unix system, as the file descriptor is kept open. This method is preferred, because even in the case of a program segfault, the file data is cleaned up by the kernel.
    • delete (or delete_on_exit) : delete file after the STXXL program exists
      This is the more conservative version of unlink, which also works on Windows. However, if the program crashes, the file is not deleted.
    • raw_device : fail if the opened path is not a raw block device.
      This flag is not required, raw devices are automatically detected.
    • queue=# : assign the disk to a specific I/O request queue and thread.
      Use this for multiple files that reside on the same physical disk.
    • devid=# : assign the disk entry a specific physical device id.
      Usually you can just omit the devid=# option, since disks are enumerated automatically. In sorting and other prefetched operations, the physical device id is used to schedule block transfers from independent devices. Thus you should label files/disks on the same physical devices with the same devid.
    • queue_length=# : specify for linuxaio the desired queue inside the linux kernel using this option.

Example: 

disk=/data01/stxxl,500G,syscall unlink
disk=/data02/stxxl,300G,syscall unlink

On Windows, one usually uses different disk drives and wincall. 

disk=c:\stxxl.tmp,700G,wincall delete
disk=d:\stxxl.tmp,200G,wincall delete

On Linux you can try to take advantage of NCQ + Kernel AIO queues: 

disk=/data01/stxxl,500G,linuxaio unlink
disk=/data02/stxxl,300G,linuxaio unlink

Recommended: File System XFS or Raw Block Devices

The library benefits from direct transfers from user memory to disk, which saves superfluous copies. We recommend to use the XFS file system, which gives good read and write performance for large files. Note that file creation speed of XFS is a bit slower, so that disk files should be precreated for optimal performance.

If the filesystems only use is to store one large STXXL disk file, we also recommend to add the following options to the mkfs.xfs command to gain maximum performance: 

$ mkfs.xfs -d agcount=1 -l size=512b

The following filesystems have been reported not to support direct I/O: tmpfs , glusterfs . By default, STXXL will first try to use direct I/O (O_DIRECT open flag). If that fails, it will print a warning and open the file without O_DIRECT.

Note
It is also possible to use raw disk devices with syscall.
Just use disk=/dev/sdb1 or similar. This will of course overwrite all data on the partitions! The I/O performance of raw disks is generally more stable and slightly higher than with file systems.
disk=/dev/sdb1,0,syscall raw_device
The raw_device flag is only for verification, STXXL will automatically detect raw block devices and also their size.

Log Files

STXXL produces two kinds of log files, a message and an error log. By setting the environment variables STXXLLOGFILE and STXXLERRLOGFILE, you can configure the location of these files. The default values are stxxl.log and stxxl.errlog, respectively.

Precreating External Memory Files

In order to get the maximum performance one can precreate disk files described in the configuration file, before running STXXL applications. A precreation utility is included in the set of STXXL utilities in stxxl_tool. Run this utility for each disk you have defined in the disk configuration file: 

$ stxxl_tool create_files <capacity> <full_disk_filename...>
// for example:
$ stxxl_tool create_files 1GiB /data01/stxxl

User-Supplied disk_config Structures

With STXXL >= 1.4.0, the library can also be configured via the user application.

All disk configuration is managed by the stxxl::config class, which contains a list of stxxl::disk_config objects. Each stxxl::disk_config object encapsulates one disk= lines from a config file, or one allocated disk.

The disk configuration must be supplied to the STXXL library before any other function calls, because the stxxl::config object must be filled before any external memory blocks are allocated by stxxl::block_manager.

int main()
{
// get uninitialized config singleton
stxxl::config * cfg = stxxl::config::get_instance();
// create a disk_config structure.
stxxl::disk_config disk1("/tmp/stxxl.tmp", 100 * 1024 * 1024, "syscall unlink");
disk1.direct = stxxl::disk_config::DIRECT_ON; // force O_DIRECT
// add disk to config
cfg->add_disk(disk1);
// add another disk
cfg->add_disk( disk_config("disk=/tmp/stxxl-2.tmp, 10 GiB, syscall unlink") );
// ... add more disks
// use STXXL library as usual ...
}