Automating backport kernel integration support
I cringe when I see a task which could be automated done manually, but complex tasks are not trivially considered possible to be automated -- to even fathom such possibilities on complex tasks at times you have to divide the work into sub tasks and eventually see if its possible to automate a series of them and which ones cannot be automated. I've had a hunch about about the prospects of fully automating Linux kernel backporting for a while now, over the years a set of advances and practices on the backports project has increased my confidence of these prospects, one of them was Increasing Automation in the Backporting of Linux Drivers Using Coccinelle SmPL [paper]. If a long paper is too much to digest, check out the Automatically backporting the Linux kernel video presentation (and if you want to learn about Coccinelle SmPL check out Julia Lawall's Coccinelle tutorial at the 2014 SUSE Labs Con) from my presentation at the 2014 SUSE Labs Conference or previous blog post about that. Towards the end of my presentation I hint at some further prospects in automation with the possibility of doing self programming of the shared backports layer targetting collateral evolutions, but I'll now review one features some folks have pestered me for a bit to incorporate: direct kernel integration with backports and which I recently completed during the 2014 SUSE Hackweek.
We now have the framework to optimize backporting collateral evolutions with the use of patches, Coccinelle SmPL grammar patches, and a shared layer. The development flow we follow helps track linux-next daily, and this reduces the amount of work when we're close to a release made by Torvalds or Greg KH. Although we make both daily linux-next based releases and also stable releases what we provide is a tarball and users and system integrators have no way of making what we provide non-modular. This is a problem for some ecosystems such as Android and ChromeOS which do not like to ship modules. You can technically take such releases, modify them somehow, and then allow integration to be able to build these drivers as built-in and although I know some folks have used this strategy before (ChromeOS was one, OpenWrt has been doing this for years) its not easy to upkeep, and update, and when a new release is made you have to re-do all the work. As of backports-20141114 we now have backports kernel integration support merged. What this means is that folks that need to stick to older kernels as base can use the backports project to do the integration of drivers from future kernels onto their kernel, with full kconfig support. You get what you expect, a new sub menu entry under 'make menuconfig' which lets you enter a submenu that lets you enable either as module / built-in device drivers / subsystems from future kernels to replace your older kernel's drivers / subsystems. The work to integrate a backports release is therefore now automated.
As you'd expect device drivers from future kernels can only be selected if the respective older driver is disabled. You can opt to compile backported drivers as modular or built-in. The ability to compile in device drivers as built-in also now enables the possibility to add support into backports features and components from the kernel which we were previously not able to backport. Integration support enables a one shot full integration support from a future release to an older release, the way to upgrade then would require simply rebasing your kernel as you bump your base kernel and doing another kernel integration when needed. If you are not rebasing your kernel in order to only upgrade to a new future set of backported drivers you can just drop the old backports/ directory and attempt a new integration with the newer release. This means you should clearly document non-upstream cherry picks on top of a backport integration, cherry pick them out and later merge them back in. This purposely favours upstream development work flow, if your cherry picks are on route upstream when you bump to a new backport you likely will drop most of the cherry picks you carry, in fact if you have policies in place to ensure they are upstream by a future release integration you'd be always striving towards 0 delta, and of course, 0 delta would imply fully automated backport work then. I hope this alone might encourage some folks to consider their own development work flows a bit, in particular those with over 6 million lines of code delta, and umm, with it taking them over 6 months to complete a rebase ;) ... On a modern laptop running the integration takes about 1-2 minutes to complete. More details are available the on the backports wiki section on backports kernel integration support. If you have any questions poke on IRC #kernel-backports on freenode or join the backports mailing list.
We now have the framework to optimize backporting collateral evolutions with the use of patches, Coccinelle SmPL grammar patches, and a shared layer. The development flow we follow helps track linux-next daily, and this reduces the amount of work when we're close to a release made by Torvalds or Greg KH. Although we make both daily linux-next based releases and also stable releases what we provide is a tarball and users and system integrators have no way of making what we provide non-modular. This is a problem for some ecosystems such as Android and ChromeOS which do not like to ship modules. You can technically take such releases, modify them somehow, and then allow integration to be able to build these drivers as built-in and although I know some folks have used this strategy before (ChromeOS was one, OpenWrt has been doing this for years) its not easy to upkeep, and update, and when a new release is made you have to re-do all the work. As of backports-20141114 we now have backports kernel integration support merged. What this means is that folks that need to stick to older kernels as base can use the backports project to do the integration of drivers from future kernels onto their kernel, with full kconfig support. You get what you expect, a new sub menu entry under 'make menuconfig' which lets you enter a submenu that lets you enable either as module / built-in device drivers / subsystems from future kernels to replace your older kernel's drivers / subsystems. The work to integrate a backports release is therefore now automated.
As you'd expect device drivers from future kernels can only be selected if the respective older driver is disabled. You can opt to compile backported drivers as modular or built-in. The ability to compile in device drivers as built-in also now enables the possibility to add support into backports features and components from the kernel which we were previously not able to backport. Integration support enables a one shot full integration support from a future release to an older release, the way to upgrade then would require simply rebasing your kernel as you bump your base kernel and doing another kernel integration when needed. If you are not rebasing your kernel in order to only upgrade to a new future set of backported drivers you can just drop the old backports/ directory and attempt a new integration with the newer release. This means you should clearly document non-upstream cherry picks on top of a backport integration, cherry pick them out and later merge them back in. This purposely favours upstream development work flow, if your cherry picks are on route upstream when you bump to a new backport you likely will drop most of the cherry picks you carry, in fact if you have policies in place to ensure they are upstream by a future release integration you'd be always striving towards 0 delta, and of course, 0 delta would imply fully automated backport work then. I hope this alone might encourage some folks to consider their own development work flows a bit, in particular those with over 6 million lines of code delta, and umm, with it taking them over 6 months to complete a rebase ;) ... On a modern laptop running the integration takes about 1-2 minutes to complete. More details are available the on the backports wiki section on backports kernel integration support. If you have any questions poke on IRC #kernel-backports on freenode or join the backports mailing list.
Comments