Discussion:
linux /dev and normal files
Elliott, Robert (Server Storage)
11 years ago
Permalink
-----Original Message-----
Of Sitsofe Wheeler
...
filename=\\.\physicaldrive1
ioengine=windowsaio
direct=1
...
size=86%
bs=4k
...

I noticed that, in linux, if you select
filename=/dev/sd<not there>
size=<something>

(e.g., if you run fio after a device has failed), it creates a
normal file of the specified size (which could be quite large
if using a script such as above).

If you use direct=1, the file is created, followed by this error:
fio: pid=8914, err=22/file:filesetup.c:611, func=open(/dev/sdad), error=Invalid argument
fio: looks like your file system does not support direct=1/buffered=0
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT

If you don't use size=, no file is created and this error
occurs:
fio: pid=0, err=22/file:filesetup.c:820, func=total_file_size, error=Invalid argument
drive_ag: you need to specify size=

If you recreate the device, udevd blows away any such file with
the block device node file, so it's not a good place to put
normal files, even if that is intentional.

In Windows, all "\\.\" paths are assumed to mean block devices,
so fio doesn't inadvertently try to create a normal file in
such a location.

Should fio treat "/dev" paths in linux the same way?

---
Rob Elli
Jens Axboe
11 years ago
Permalink
...
This is because /dev is usually devtmpfs these days, and that fs does
not support O_DIRECT.
...
No, I don't think so. It's a directory like anything else. Who knows
what will happen in the future, with all the core OS changes in Linux.
/dev could be a symlink, or special files could be in a different
location completely. What about FreeBSD, should we do the same there?
Or Solaris?

I'd much rather keep fio ignorant of any "special" directories, even if
it means that sometimes you do potentially run into issues like the
above, where you specify a block device that does not exist.
--
Jens Axboe
Elliott, Robert (Server Storage)
11 years ago
Permalink
-----Original Message-----
...
I'd much rather keep fio ignorant of any "special" directories, even if
it means that sometimes you do potentially run into issues like the
above, where you specify a block device that does not exist.
How about an option in the script:
direct=2 file must exist and be a block device, otherwise skip it

If the size is too big, this is the error:

drive_r: Laying out IO file(s) (1 file(s) / 1572864000MB)
fio: posix_fallocate fails: No space left on device

fio: ENOSPC on laying out file, stopping
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
fio: pid=8991, err=22/file:filesetup.c:611, func=open(/dev/sdnothere), error=Invalid argument
fio: pid=8990, err=22/file:filesetup.c:611, func=open(/dev/sdnothere), error=Invalid argument

and it leaves a "file" that looks like:

$ ls -alt /dev/sd*
-rw-r--r-- 1 root root 1649267441664000 Sep 30 20:
Jens Axboe
11 years ago
Permalink
Post by Elliott, Robert (Server Storage)
-----Original Message-----
...
I'd much rather keep fio ignorant of any "special" directories, even if
it means that sometimes you do potentially run into issues like the
above, where you specify a block device that does not exist.
direct=2 file must exist and be a block device, otherwise skip it
Might be a bit nasty to overload direct= like that. But I'd be open to
adding an option that says must_be_bdev or something, which can be a
bool as well.
...
Yeah I know, I've had that happen myself 1-2 times.
--
Jens Axboe
Jens Axboe
11 years ago
Permalink
...
Totally untested patch attached (it compiles). There's a new option,
filetype. If not set, if will allow any file type. Can be set to:

any default, allow any file type
file regular file
bdev block device
chardev character device
pipe pipe

So for your case, you'd set

filetype=bdev

and you'd avoid the issue. Note that it only works for files specified
with filename=. Any other way of adding a file (blktrace replay,
opendir=, or fio added files when no filename= given) will ignore the
filetype setting.
--
Jens Axboe
Sitsofe Wheeler
11 years ago
Permalink
...
How about a create parameter which defaults to create=1 giving the
current behaviour and create=0 where the job fails if the file doesn't
already exist? It won't catch as much but should be portable.
--
Sitsofe | http://sucs.org/~sits/
Jens Axboe
11 years ago
Permalink
...
Agree, that will be both easier to maintain and probably just as useful
as the more expanded options. Lets call it 'allow_file_create'.
--
Jens Axboe
Elliott, Robert (Server Storage)
11 years ago
Permalink
...
...
Elliott, Robert (Server Storage)
11 years ago
Permalink
...
Tested with 16 devices sdr to sdag, where sdaf and sdag are
offline.

It prints a message about sdaf and exits, and does not
create a normal file in that location:
parse 11480 [drive_af]
parse 11480 Parsing section [drive_af]
file 11480 dup files: 0
parse 11480 filename=/dev/sdaf
parse 11480
parse 11480 [drive_ag]
parse 11480 handle_option=filename, ptr=/dev/sdaf
parse 11480 __handle_option=filename, type=5, ptr=/dev/sdaf
file 11480 add file /dev/sdaf
file 11480 resize file array to 1 files
fio: dropping file '/dev/sdaf' of wrong type
fio: wanted bdev, got file
fio: failed parsing filename=/dev/sdaf
fio: job drive_af dropped
parse 11480 free options
parse 11480 free options
io 11480 ioengine cpuio unregistered
io 11480 ioengine mmap unregistered
io 11480 ioengine sync unregistered
...

compared to a bdev that exists:
parse 11480 handle_option=filename, ptr=/dev/sdae
parse 11480 __handle_option=filename, type=5, ptr=/dev/sdae
file 11480 add file /dev/sdae
file 11480 resize file array to 1 files
file 11480 file 0x7f7903703a90 "/dev/sdae" added at 0
io 11480 load ioengine libaio
parse 11480 init options
drive_ae: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=96
file 11480 dup files: 1
io 11480 load ioengine libaio

That is because the do {} while (!ret) loop in
__parse_jobs_ini gives up on the first add_job failure.
add_job calls parse_options which calls handle_option
which fails.

It might be better to continue to attem
Jens Axboe
11 years ago
Permalink
...
We should just go for the simpler create/nocreate solution, I prefer
that. This is too involved, and it's hard to get all the cases right
since there are a bunch of different ways that files are added. Which
was why this patch only focused on filename=, since that is the one that
is the most interesting. But it still isn't pretty. If we add the
allow_file_create option, it's basically a two-liner change in a few
places to not use O_CREAT.
--
Jens Axboe
Jens Axboe
11 years ago
Permalink
...
That took roughly 4 minutes... Does this work? Set allow_file_create=0
to disallow creating new files.
--
Jens Axboe
Elliott, Robert (Server Storage)
11 years ago
Permalink
...
Mostly.

With no size= setting or size=xx%, the error messages it prints are
about the lack of size, not the fact that file creation is disallowed.
It does proceed to run the jobs that have valid filenames.

drive_af: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:822, func=total_file_size, error=Invalid argument
drive_af: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:822, func=total_file_size, error=Invalid argument
...
drive_ag: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:822, func=total_file_size, error=Invalid argument
Jobs: 168 (f=168): [r(168),X(24)] [0.0% done] [2449MB/0KB/0KB /s] [627K/0/0 iops] [eta 02d:11h:58m:33s]


With size=xxm (a fixed size), it prints complaints about O_DIRECT.
It does proceed to run the jobs that have valid filenames.

fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
fio: pid=21834, err=22/file:filesetup.c:613, func=open(/dev/sdaf), error=Invalid argument
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
fio: pid=21846, err=22/file:filesetup.c:613, func=open(/dev/sdag), error=Invalid argument
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
...
fio: pid=21856, err=22/file:filesetup.c:613, func=open(/dev/sdag), error=Invalid argument
Jobs: 168 (f=168): [r(168),X(24)] [0.0% done] [2449MB/0KB/0KB /s] [627K/0/0 iops] [eta 02d:11h:58m:50s]




---
Rob Elliott HP Server St
Jens Axboe
11 years ago
Permalink
...
Fio always does this, if the file does not exist and no size is given.
The reason is that it doesn't know what size to create. But we can
augment that a little bit if allow_file_create == 0, since the size
setting would not change that.
...
OK, let me test that, will spin a new one.
--
Jens Axboe
Jens Axboe
11 years ago
Permalink
...
This works for me - though I cannot reproduce your O_DIRECT warning
above, what was run to generate that?
--
Jens Axboe
Continue reading on narkive:
Loading...