Splitting the data directory

From Bitcoin Wiki
Jump to: navigation, search

Bitcoin Core normally puts all of its data into one data directory, but oftentimes it is useful to adjust things so that certain of these files go elsewhere.

If your data directory is on a magnetic disk: Moving chainstate for improved speed

Bitcoin Core's overall speed is significantly affected by the random-access speed of the contents of the chainstate directory; if your data directory is located on a magnetic disk, your chainstate access speed will very probably be the biggest performance bottleneck. You can therefore often massively improve performance by moving just the chainstate directory (which is only a few GB) to an SSD drive. Even moving the chainstate directory to a USB flash drive can often provide a large performance improvement, especially if the USB flash drive is advertised as being particularly high-speed.

To move it:

  1. Shut down Bitcoin Core
  2. From the data directory, cut the chainstate directory (not just its contents, but the directory itself) and paste it somewhere on the target drive. On Linux, make sure that permissions allow for Bitcoin Core to read/write the directory and its files at the new location.
  3. Open a terminal in the data directory. (On Windows, Shift-RightClick the data directory and choose "open a command window here".) On Linux, run ln -s /absolute/path/to/chainstate. For example, if you moved your chainstate so that its new location is /mnt/ssd/core/chainstate (ie. within that directory are a bunch of ldb files), you'd run ln -s /mnt/ssd/core/chainstate from immediately within the data directory. On Windows, run mklink /D chainstate ?:\path\to\chainstate. For example, if you moved your chainstate so that its new location is F:\core\chainstate (ie. within that directory are a bunch of ldb files), you'd run mklink /D chainstate F:\core\chainstate from immediately within the data directory.

If you're using a removable drive, make sure that the path to the real chainstate does not change.

If your data directory is on an SSD: Moving the blocks database to save space

Unlike the chainstate directory mentioned in the previous section, the blk* and rev* files in the blocks directory in the data directory are rarely accessed, and when accessed they are accessed in a highly sequential access pattern, so it will not affect performance much at all to move them to a magnetic disk, thereby freeing up 100+ GB on your SSD.

Note that you can avoid the need to store the blocks database at all by running Bitcoin Core with pruning enabled (ie. add prune=550 to bitcoin.conf), but then you will lose some features and be slightly less helpful to the network. Security is not affected by this, however, and you remain a full node.

Unlike in the chainstate case, you don't want to move the entire blocks directory because this would move the performance-relevant blocks/index directory as well. Instead, you can place the following script in the blocks directory and run it periodically with Bitcoin Core shut down to move the blk* and rev* files individually. Do not run the script while Bitcoin Core is running or you will probably wreck things in such a way that you will have to sync from 0. Modify BLK_TARGET=... in the script to point to your desired target location.

Linux:

#!/bin/bash
set -e
BLK_TARGET=/mnt/ssd/core/blocks #Replace with your destination, no trailing slash

find . -name '*.dat' -type f -printf '%f\n' > tomove
while read line; do
        echo $line
        mv "$line" "$BLK_TARGET/$line"
        ln -s "$BLK_TARGET/$line" "$line"
done <tomove
rm tomove
echo Done

Windows (save as move.bat or something):

@echo off
REM Change the following line to your destination, no trailing slash
SET BLK_TARGET=F:\core\blocks
for /f %%f in ('dir /A-L /b *.dat') do (
	move %%f %BLK_TARGET%\%%f
	if exist %%f (
		echo Something went wrong
		pause
		exit
	)
	mklink %%f %BLK_TARGET%\%%f
)
echo Done
pause

These scripts expect to be inside of the blocks directory in the data directory, next to the blk and rev files. They do only minimal error handling -- use with care.

To undo the script, just move all of the files back to where they were originally, overwriting the symlinks.