An Attempt at Reducing Costs of Disk I/O in Go

为了实现磁盘上的IO，go语言使用一组IO线程池用来提高CPU的使用率

本文是在go语言上尝试了接入了linux libaio，并没有发现性能的提升。

使用O_DIRECT意味着数据直接到达磁盘上，这将不能有效的利用pagecache用来加速，合并读写请求；O_DIRECT的方式对读写的大小有限制，必须是block size大小对齐(4k)，而对于普通用户来讲是需要传统的字节对齐，带有pagecache缓存的API接口
可以在用户态做一层缓存，这样也会带来一些问题，pagecache是系统全局的，而如果在用户态做缓存，
每个进程需要使用自己独立的缓存，内存的消耗会比pagecache大，
缓存机制可能不如全局的pagecache看到的冷热指标更多，效果可能会有损失；
每次进程重启的时候，之前的用户态缓存就会失效，冷启动时间较长

"In order to facilitate disk I/O, Go keeps a pool of I/O threads around to improve CPU utilization while threads block during I/O."

"While we are unable to see improved performance, we believe this is due to the limited support for AIO provided by the Linux kernel"

"Green threads provide the benefit of reduced overheads present in kernel threads such as the cost of entering the kernel to perform context switches, wasted memory when reserving a whole page for stack, and green threads introduce faster alternatives to futex wait"

"The requesting Goroutines will block until the scheduler has completed their request, but the M that Goroutine is blocking on will take another G from its work queue (or steal one) and continue execution. In order to perform disk operations and continue to receive requests, our scheduler will utilize Linux Asynchronous I/O (AIO) system calls to batch requests and allow the kernel to process them in the background"

"AIO is actually implemented to be blocking due to missing functionality in Linux"

"Opening files with O DIRECT means that access to files now goes directly to disk. This implies that subsequent disk operations on the same block will perform poorly since we will no longer be using kernel disk caching mechanisms. Even worse, reads and writes must be aligned to disk block boundaries and a multiple of disk block size"

"Based on our original goal of modifying IO in Go in a user transparent manner, direct disk access is not an option, neither performance nor read/write semantics match Go's current IO API. Go users expect the traditional, cached (highly performant), byte granular operations provided by the current API"

"To summarize our results, we were unable to improve the performance of scheduling disk IO requests in Go. We believe this is not due to the costs introduced by our AIO scheduler but is due to non-existent kernel support for AIO VFS operations. The restriction to direct disk access means we are unable to modify Go in a transparent manner, direct access would introduce massive performance decreases on re-access of data, and users would not be able to modify files a byte granularity."

"we also see the possibility of layering a disk caching library on top of our scheduler in order to continue using the existing AIO in the Linux kernel while avoiding the uncached performance hits. However, there are quite a few issues with this approach. First, each Go program will now be managing disk caches so they each will be wasting their own program memory space. Second ever time a program starts, it will take time for the disk cache to repopulate. General purpose disk caching is better executed globally in the kernel, so adding caching on top of our scheduler is also not a general purpose solution"

原文链接： https://www.cnblogs.com/zhaojiangkun/p/17407131.html