Date
1 - 20 of 25
[RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
|
Kyungsik Lee <kyungsik.lee@...>
Hi,
First of all, Thank you for the comments and emails from the community. Here is the second version of support for LZ4-compressed kernel. In this version, lz4_decompress() has been added. In case of knowing the uncompressed data size, this function can be used to decompress more faster. Through the benchmark, it was found that -Os Compiler flag for decompress.o brought better decompression performance in most of cases (ex, different compiler and hardware spec.) in ARM architecture. Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best option even though it is supported. The decompression speed can be slightly slower in some cases. This patchset is based on 3.8. Any comments are appreciated. Thanks, Kyungsik Benchmark Results(PATCH v2) Compiler: Linaro ARM gcc 4.6.2 1. ARMv7, 1.5GHz based board Kernel: linux 3.4 Uncompressed Kernel Size: 14MB Compressed Size Decompression Speed LZO 6.7MB 21.1MB/s LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA) 2. ARMv7, 1.7GHz based board Kernel: linux 3.7 Uncompressed Kernel Size: 14MB Compressed Size Decompression Speed LZO 6.0MB 34.1MB/s LZ4 6.5MB 86.7MB/s UA: Unaligned memory Access support Change log: v2 - Clean up code - Enable unaligned access for ARM v6 and above with CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - Add lz4_decompress() for faster decompression with uncompressed output size - Use lz4_decompress() for LZ4-compressed kernel during boot-process - Apply -Os to decompress.o to improve decompress performance during boot-up process Kyungsik Lee (4): decompressor: Add LZ4 decompressor module lib: Add support for LZ4-compressed kernel arm: Add support for LZ4-compressed kernel x86: Add support for LZ4-compressed kernel arch/arm/Kconfig | 1 + arch/arm/boot/compressed/.gitignore | 1 + arch/arm/boot/compressed/Makefile | 6 +- arch/arm/boot/compressed/decompress.c | 4 + arch/arm/boot/compressed/piggy.lz4.S | 6 + arch/x86/Kconfig | 1 + arch/x86/boot/compressed/Makefile | 5 +- arch/x86/boot/compressed/misc.c | 4 + include/linux/decompress/unlz4.h | 10 + include/linux/lz4.h | 48 +++++ init/Kconfig | 13 +- lib/Kconfig | 7 + lib/Makefile | 2 + lib/decompress.c | 5 + lib/decompress_unlz4.c | 190 +++++++++++++++++++ lib/lz4/Makefile | 1 + lib/lz4/lz4_decompress.c | 331 ++++++++++++++++++++++++++++++++++ lib/lz4/lz4defs.h | 93 ++++++++++ scripts/Makefile.lib | 5 + usr/Kconfig | 9 + 20 files changed, 739 insertions(+), 3 deletions(-) create mode 100644 arch/arm/boot/compressed/piggy.lz4.S create mode 100644 include/linux/decompress/unlz4.h create mode 100644 include/linux/lz4.h create mode 100644 lib/decompress_unlz4.c create mode 100644 lib/lz4/Makefile create mode 100644 lib/lz4/lz4_decompress.c create mode 100644 lib/lz4/lz4defs.h -- 1.8.1.1 |
|
|
|
Markus F.X.J. Oberhumer <markus@...>
On 2013-02-26 07:24, Kyungsik Lee wrote:
Hi,Did you actually *try* the new LZO version and the patch (which is attached once again) as explained in https://lkml.org/lkml/2013/2/3/367 ? Because the new LZO version is faster than LZ4 in my testing, at least when comparing apples with apples and enabling unaligned access in BOTH versions: armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size: compression speed decompression speed LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access ~Markus
-- Markus Oberhumer, <markus@...>, http://www.oberhumer.com/ |
|
|
|
Nicolas Pitre <nico@...>
On Tue, 26 Feb 2013, Markus F.X.J. Oberhumer wrote:
On 2013-02-26 07:24, Kyungsik Lee wrote:To be fair, you should also take into account the compressed size of aHi,Did you actually *try* the new LZO version and the patch (which is attached typical ARM kernel. Sometimes a slightly slower decompressor may be faster overall if the compressed image to work on is smaller. Nicolas |
|
|
|
Peter Korsgaard
Hi,"Nicolas" == Nicolas Pitre <nico@...> writes: >> Did you actually *try* the new LZO version and the patch (which is attached >> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ? >> >> Because the new LZO version is faster than LZ4 in my testing, at least >> when comparing apples with apples and enabling unaligned access in >> BOTH versions: >> >> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size: >> >> compression speed decompression speed >> >> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access >> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access >> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access Nicolas> To be fair, you should also take into account the compressed Nicolas> size of a typical ARM kernel. Sometimes a slightly slower Nicolas> decompressor may be faster overall if the compressed image to Nicolas> work on is smaller. Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the introduction mail: 1. ARMv7, 1.5GHz based board Kernel: linux 3.4 Uncompressed Kernel Size: 14MB Compressed Size Decompression Speed LZO 6.7MB 21.1MB/s LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA) -- Bye, Peter Korsgaard |
|
|
|
Nicolas Pitre <nico@...>
On Tue, 26 Feb 2013, Peter Korsgaard wrote:
OK. If LZO is now faster than LZ4 while still compressing more then IHi,"Nicolas" == Nicolas Pitre <nico@...> writes: have no argument. Nicolas |
|
|
|
Russell King - ARM Linux <linux@...>
On Tue, Feb 26, 2013 at 10:58:02PM +0100, Peter Korsgaard wrote:
Well, until someone can put all the pieces together so that a reasonablyHi,"Nicolas" == Nicolas Pitre <nico@...> writes: meaningful test between: - The new LZO code - The new LZ4 code then you're all comparing different things. TBH, I'm disappointed that all the comments about this from the previous posting of LZ4 have been totally ignored, and we _still_ don't really have this information. It seems like replying to these threads is a waste of time. So... for a selected kernel version of a particular size, can we please have a comparison between the new LZO code and this LZ4 code, so that we can see whether it's worth updating the LZO code or replacing the LZO code with LZ4? |
|
|
|
Joe Perches <joe@...>
On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
So... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? |
|
|
|
Kyungsik Lee <kyungsik.lee@...>
On Tue, Feb 26, 2013 at 09:33:22PM +0100, Markus F.X.J. Oberhumer wrote:
On 2013-02-26 07:24, Kyungsik Lee wrote:I agree that the new LZO version provided shows better decompressionHi,Did you actually *try* the new LZO version and the patch (which is attached speed than 3.7 based. It is much improved especially for UA. Compiler: Linaro ARM gcc 4.6.2 2. ARMv7, 1.7GHz based board Kernel: linux 3.7 Uncompressed Kernel Size: 14MB Compressed Size Decompression Speed LZO 6.0MB 34.1MB/s Old ---------------------------------------- 6.0MB 34.7MB/s New 6.0MB 52.2MB/s(UA) ============================================= LZ4 6.5MB 86.7MB/s UA: Unaligned memory Access support One thing I can say that the code you may have used, guessing "lz4demo" is not the same code provided in this patch. It has been ported for the kernel and uses different function not like the "lz4demo". Thanks, Kyungsik |
|
|
|
Russell King - ARM Linux <linux@...>
On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
Compiler: Linaro ARM gcc 4.6.2That is pretty conclusive - it shows an 8% increase in image size vs a 66% increase in decompression speed. It will take a _lot_ to offset that increase in decompression speed. So, what I think is that yes, we should accept LZ4 and drop LZO from the kernel - the "fast but may not be small" compression title has clearly been taken by LZ4. Akpm - what's your thoughts? |
|
|
|
Russell King - ARM Linux <linux@...>
On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:Please read the comments against the previous posting of these patchesSo... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? where I first stated this argument - and with agreement from those following the thread. The thread started on 26 Jan 2013. Thanks. |
|
|
|
Johannes Stezenbach <js@...>
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:I think LZO may be used by squashfs, jffs2 and btrfs, thus youCompiler: Linaro ARM gcc 4.6.2That is pretty conclusive - it shows an 8% increase in image size vs a cannot drop it without breaking on disk storage formats. Johannes |
|
|
|
Kyungsik Lee <kyungsik.lee@...>
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:I have read the comments regarding how many compressors the kernelCompiler: Linaro ARM gcc 4.6.2That is pretty conclusive - it shows an 8% increase in image size vs a should support and understand that it can not support all the compressors available. However, I don't think that LZO can be replaced by LZ4 in all the cases. The benchmark above shows only about improved decompression speed. Thanks, Kyungsik |
|
|
|
Nicolas Pitre <nico@...>
On Wed, 27 Feb 2013, Johannes Stezenbach wrote:
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:It is not about dropping LZO from the kernel entirely. It's aboutOn Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:I think LZO may be used by squashfs, jffs2 and btrfs, thus youCompiler: Linaro ARM gcc 4.6.2That is pretty conclusive - it shows an 8% increase in image size vs a removing support for compressing zImage using LZO (and some others). There is no compatibility issue as zImage embeds its own decompression code. Nicolas |
|
|
|
Joe Perches <joe@...>
On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:https://lkml.org/lkml/2013/1/29/145On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:Please read the comments against the previous posting of these patchesSo... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? I did not and do not see significant value in adding LZ4 given Markus' LZO improvements. I asked about LZO. Why would the LZO code not be updated? The new LZO code is faster than ever and it's a standalone improvement. Markus has posted what seems a clean git pull request. It was not cc'd to arm or linux-arch. http://linux-kernel.2935.n7.nabble.com/GIT-PULL-Update-LZO-compression-code-for-v3-9-td605184.html |
|
|
|
Nicolas Pitre <nico@...>
On Wed, 27 Feb 2013, Joe Perches wrote:
https://lkml.org/lkml/2013/1/29/145Connecting to lkml.org (lkml.org)|87.253.128.182|:443... connected. HTTP request sent, awaiting response... 500 Server Error I did not and do not see significant value inPlease someone post a comprehensive comparison with all the results in the same email. The new LZO code is faster than ever and it'sIt is used by filesystems, etc. So of course it needs to be updated to faster code. Markus has posted what seems a clean git pullMaybe a reminder should be sent to Linus about this. From the above we can see this: **LZO-2013-UA : updated LZO version available in linux-next plus experimental ARM Unaligned Access patch. This needs approval from some ARM maintainer ist NOT YET INCLUDED. What is that experimental patch in need of approval? Nicolas |
|
|
|
Russell King - ARM Linux <linux@...>
On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:Sorry, a 66% increase in decompression speed over the updated LZO codeOn Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:https://lkml.org/lkml/2013/1/29/145On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:Please read the comments against the previous posting of these patchesSo... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? isn't "significant value" ? I'm curious - what in your mind qualifies "significant value" ? Maybe "significant value" is a patch which buggily involves converting all those "<n>" printk format strings in assembly files to KERN_* macros, thereby breaking those strings because you've not paid attention to what .asciz means? (Yes, I've just cleaned that crap up after you...) Why would the LZO code not be updated?I'm not saying that the LZO code should not be updated. I'm saying that the kernel boot time decompressor is not a play ground for an ever increasing number of "my favourite compression method" crap. We don't need four, five or even six compression methods there. We just need three - a "fast but large", "small but slow" and "all round popular medium". |
|
|
|
Borislav Petkov <bp@...>
On Wed, Feb 27, 2013 at 04:31:18PM +0000, Russell King - ARM Linux wrote:
I'm not saying that the LZO code should not be updated. I'm sayingHell yeah! -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- |
|
|
|
Joe Perches <joe@...>
On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:We disagree.On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:Sorry, a 66% increase in decompression speed over the updated LZO codeOn Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:https://lkml.org/lkml/2013/1/29/145On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:Please read the comments against the previous posting of these patchesSo... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? I'm curious - what in your mind qualifies "significant value" ?faster boot time. smaller, faster overall code. Maybe "significant value" is a patch which buggily involves convertingIf you mean commit 0cc41e4a21d43, perhaps you could clarify with an example. I don't see any relevant changes by you in -next, but maybe I'm not looking in the right spot. The change did enable reducing code size. You said:Why would the LZO code not be updated?I'm not saying that the LZO code should not be updated. Sounded as if you were doubtful to me.so that we can see whether it's worth updating the LZO code I'm saying thatCompletely agree. |
|
|
|
Nicolas Pitre <nico@...>
On Wed, 27 Feb 2013, Joe Perches wrote:
On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:Sorry, but you certainly successfully got me confused, and probablyOn Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:We disagree.On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:Sorry, a 66% increase in decompression speed over the updated LZO codeOn Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:https://lkml.org/lkml/2013/1/29/145On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:Please read the comments against the previous posting of these patchesSo... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? others as well. RMK says that "66% increase in decompression speed over LZO" is significant. You apparently disagree with that. Then you say that faster boot time is significant. Again, can you (or anyone else) provide comprehensive test results in a single email with both compression methods? Nicolas |
|
|
|
Russell King - ARM Linux <linux@...>
On Wed, Feb 27, 2013 at 09:04:48AM -0800, Joe Perches wrote:
On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:ROTFL.On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:We disagree.On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:Sorry, a 66% increase in decompression speed over the updated LZO codeOn Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:https://lkml.org/lkml/2013/1/29/145On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:Please read the comments against the previous posting of these patchesSo... for a selected kernel version of a particular size, can we pleaseHow could it be questionable that it's worth updating the LZO code? ROTFL again! Because you've just disagreed with your above statement.I'm curious - what in your mind qualifies "significant value" ?faster boot time. smaller, faster overall code. "66% increase in decompression speed" as far as I know _is_ "faster boot time" ! While recently asking someone to enable VFP debugging, so I could helpMaybe "significant value" is a patch which buggily involves convertingIf you mean commit 0cc41e4a21d43, perhaps you could clarify with an sort out a problem they had reported, this is the debug output I was greeted by thanks to your meddling: [ 927.235546] \x01\x01\x01\x01\x01\x01\x01\x01 ... [ 927.241505] \x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01 Yes, really useful debug output isn't it? You can really see what's going on there. These are coming from ultimately two commits - the one you refer to above, which on its own would've changed the printk string to be merely "<7>" - and the follow on commit changing the way printk levels are dealt with. The above output is produced by: #define KERN_SOH "\001" /* ASCII Start Of Header */ #define KERN_DEBUG KERN_SOH "7" /* debug-level messages */ .asciz KERN_DEBUG "VFP: \str\n" 7.6 `.asciz "STRING"'... ======================== `.asciz' is just like `.ascii', but each string is followed by a zero ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ byte. The "z" in `.asciz' stands for "zero". ^^^^^ 0000 01003700 5646503a 20696e73 74722025 ..7.VFP: instr % ^^ ^^ 0010 30387820 70632025 30387820 73746174 08x pc %08x stat 0020 65202570 0a000100 37005646 503a2066 e %p....7.VFP: f ^^ ... That is: \x01 \x00 7 \x00 VFP: instr %08x pc %08x state %p \x00 See - three separately terminated strings because you changed: .asciz "<7>VFP: \str\n" to: .asciz "<7>" "VFP: \str\n" which turned it into _two_ separately NUL-terminated strings, and then the follow-on changes to printk kern levels changed this to: .asciz "\001" "7" "VFP: \str\n" producing _three_ separately NUL-terminated strings. The commit is not in mainline, nor linux-next, but in my tree as of yesterday (e36815e2e), ready to be pushed out when I've finished working on fixing other problems with VFP - or when I decide to push it out ready for submission during this merge window. The change did enable reducing code size.??? Yea, right, meanwhile breaking the ability of stuff to produce kernel messages. _In_ the decompressor. We're talking about the _decompressor_ inYou said:Why would the LZO code not be updated?I'm not saying that the LZO code should not be updated.Sounded as if you were doubtful to me.so that we can see whether it's worth updating the LZO code this thread. |
|
|