This week in KDE: moar performance!

Thom Holwerda 2020-04-08 KDE 9 Comments

Some very nice performance fixes landed this week, which should substantially boost move and copy speeds for local transfers and transfers to and from Samba shares in particular. But that’s not all, and there’s more on the menu…

Every week, there’s a blog post highlighting the various changes, bugfixes, small new features, fixed paper cuts, and other small changes within KDE and its associated projects and programs. They’re a joy to read, and I would love it if more major software projects did this.

About The Author

Thom Holwerda

Follow me on Mastodon @[email protected]

9 Comments

2020-04-08 7:28 pm

cb88
The problem with KDE is they add bloat faster than they improve performance…

2020-04-09 10:05 am

avgalen
Actually accurate!
https://bugs.kde.org/show_activity.cgi?id=291835
It took them over a year, from 2012-01-21 13:26:59 UTC to 2013-02-20 22:11:36 UTC to confirm this and 7 more years to fix it. Apparently in that time everybody suffered from transfer rates over SMB that were 20 MB/sec instead of 90 MB/sec when using SMB????
2020-04-10 6:43 pm

richarson
Not my experience, it’s getting better with every release.

2020-04-09 9:57 am

avgalen
I am confused. Isn’t KDE a windowing/desktop environment? What does move/copy speed have to do with the GUI? Unless they build a ridiculously dumb progressbar that slows the actual file operation down I really don’t understand

2020-04-10 8:05 pm

Alfman verbose=1
avgalen,

I am confused. Isn’t KDE a windowing/desktop environment? What does move/copy speed have to do with the GUI? Unless they build a ridiculously dumb progressbar that slows the actual file operation down I really don’t understand

I’ve noticed that when I copy large files to samba shares in particular under KDE/dolphin, I would get bad transfer speeds. I’ve never looked into what was causing it before, however I encountered this many times. Apparently it was only copying 32kb at a time. The update sets it to 512k. I haven’t updated yet however I believe that it could help a lot.
2020-04-11 10:59 am

friedchicken
Great question avgalen. I can’t think of any reason why a GUI would impact copy speeds. A GUI shouldn’t have its hands on the actual copying at all. Even a progress bar that frequently polls how much has been copied shouldn’t affect it. This reminds me of Windows 95 and the fact that moving the mouse around made i/o faster. Maybe doing that will help KDE users.

2020-04-11 6:46 pm

Alfman verbose=1
friedchilden,

Great question avgalen. I can’t think of any reason why a GUI would impact copy speeds. A GUI shouldn’t have its hands on the actual copying at all.

It’s not about the GUI so much as it is about the lack of a copy function in POSIX & linux. For better or worse file copy is implemented in the application that’s doing the copying.

https://stackoverflow.com/questions/17666316/is-there-a-posix-function-to-copy-a-file

You can’t find the utility function for copying a file because there isn’t a need for it in the same way; it can be built out of ‘spare parts’. Functions like unlink() and symlink() can’t be built in terms of other functions, whereas functions such as a hypothetical copy_file() can (so you must).

Even the cp command…

I tried running strace on the cp command on Linux and it actually open both files and it reads from one file and write to the other one in blocks of 32768 bytes:

…
stat64(“log”, {st_mode=S_IFREG|0644, st_size=352, …}) = 0
stat64(“copied”, 0xbf99e1c0) = -1 ENOENT (No such file or directory)
open(“log”, O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=352, …}) = 0
open(“copied”, O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0644) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=0, …}) = 0
read(3, “2012-04-27 22:26:55-0400 [-] Log”…, 32768) = 352
write(4, “2012-04-27 22:26:55-0400 [-] Log”…, 352) = 352
read(3, “”, 32768) = 0
close(4) = 0
close(3) = 0
…

So yes, there is no cp syscall out there. Hope it helps you.

(On my system eight years later cp is using a 128k buffer instead of 32k).

It would make more sense to use the sendfile syscall, but that’s unlikley to make much difference for a copy process that is obviously much more IO bound than CPU bound.
https://linux.die.net/man/2/sendfile

Small buffers will significantly amplify the number of read/write requests. This doesn’t hurt things too badly on the local disk operations because linux applies both read-ahead and write caching, but with network shares the added latency makes everything crawl. The solution was so simple, IMHO it’s ridiculous this wasn’t fixed long ago.

However, I speculate one reason they didn’t could be because linux/posix don’t offer an API to monitor the progress of an ongoing transfer. Without an API, small buffers provide more frequent updates while large buffers reduce overhead. If you want a “smooth” progress indicator even on slow media & networks then using small buffers would have been the only way to guarantees frequent status updates from the kernel. For example, had they used a 512KB buffer size then, it could take over half a minute to transfer a buffer of data over a modem or floppy disk, making for a terrible progress indicator. These interfaces are of course obsolete, but my guess is that maybe it could have factored into the original design.

2020-04-12 11:49 am

friedchicken
Egoman,

“The solution was so simple, IMHO it’s ridiculous this wasn’t fixed long ago.”

Agreed, especially considering how common network shares are. You would think this kind of issue would be made more of a priority. I’m curious to know why it wasn’t.
2020-04-14 5:48 am

avgalen
Thanks for all that research. So the lack of kernel-level functionality indeed made it necessary for higher level frameworks like KDE to implement their own IO-functionality (KIO) and of course introduce their own bugs (sorry “1990’s level choices”). So thanks to this article and the comment section I learned a lot