efficiency of fwrite for massive numbers of small writes

I have a program that saves many large files >1GB using fwrite It works fine, but unfortunately due to the nature of the data each call to fwrite only writes 1-4bytes. with the result that the write can take over an hour, with most of this time seemingly due to the syscall overhead (or at least in the library function of fwrite). I have a similar problem with fread.

Does anyone know of any existing / library functions that will buffer these writes and reads with an inline function, or is this another roll your own?

Solution

If you write from just one thread, try using fwrite_unlocked. It does wonders relative to straight fwrite in this kind of scenarios.

Answered By — Eugene Ryabtsev

Answer Checked By — David Goodson (FixIt Volunteer)

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0