CELF Project Proposal- Refactoring Qi, lightweight bootloader


Matt Hsu <matt@...>
 

Summary: Refactoring Qi, lightweight bootloader.

Proposer: Matt Hsu


Description:

Qi (named by Alan Cox on Openmoko kernel list) is a minimal bootloader that
"breathes life" into Linux. Its goal is to stay close to the minimum needed
to "load" and then "boot" Linux -- no boot menus, additional peripheral init
or private states.

Qi currently supports samsung s3c24xx series, s3c6410, TI omap3530. The
more support platforms are planned to add on. The purpose of this project
would be to improve Qi's maintainability, portability. Ideally, this would
make people spend less time on bootloader development but be more focus
on Linux system.

Project objectives:

- Make the hierarchy of source files more sensible and clean

- Generalize components which could be used in common such as I2C drivers.
Example: platform specific I2C driver -> GPIO bitbang driver.

- Remove duplicated, unused code, header definition. Keep Qi as minimum as needed.

Related work:

http://wiki.openmoko.org/wiki/Qi

Development branches are hosted here:

http://git.warmcat.com/cgi-bin/cgit/qi/
http://gitorious.org/0xlab-bootloader

Scope:
Unknown.

Thanks.
Matt


Andy Green <andy@...>
 

On 12/17/09 08:31, Somebody in the thread at some point said:

Hi -

Just read about this a little late :-)

I wrote the bulk of Qi while at Openmoko.

Qi currently supports samsung s3c24xx series, s3c6410, TI omap3530. The
I also ported it to iMX31.

- Generalize components which could be used in common such as I2C drivers.
Example: platform specific I2C driver -> GPIO bitbang driver.
I wrote a generic bitbang I2C "driver" for Qi back in 2007, it's used on the GTA02 build IIRC to talk to the PMU:


http://git.warmcat.com/cgi-bin/cgit/qi/tree/src/drivers/i2c-bitbang.c?h=txtr

- Remove duplicated, unused code, header definition. Keep Qi as minimum
as needed.
What I can suggest would be worthwhile goals are:

- extending the targeted CPU arches

- folding the OMAP branch (written by Matt) back into the main tree

- improving the memory test to force burst mode

On the txtr branch (currently they are my customer and we use Qi on their iMX31 e-book reader) there's already various tidying up and advances like FAT filesystem, PRNG-based memory test.

http://git.warmcat.com/cgi-bin/cgit/qi/log/?h=txtr

Here is a bit more about the reasoning for bothering with another bootloader and the philosophy behind Qi.

While working at Openmoko maintaining their kernel, it became obvious that U-Boot was turning into a mini-me for Linux. Many cut-down Linux drivers were appearing there, there was a shell with an environment holding private states like booting action commands. This led to a situation where two identical devices with same patchlevel of everything including bootloader may act differently according to what's in that hidden environment. Wanting a "boot menu" meant supporting graphics and some of the drivers like for Glamo were complex and always forked from their Linux counterparts.

The burden of maintaining forked drivers in two places (the U-Boot ones had to be "dumbed down" to the point it can't share sources) and the additional complexity of trying to manage power and additional device state in two places are all actually completely unnecessary. All the bootloader needs is just enough to get hold of a kernel and boot it. That is Qi's philosophy in a nutshell: "load [the kernel], then boot". Linux is then the single place for good support of all assets on the device.

It can work from NAND or NOR but actually the existing support is designed to work with SD Card boot. Recent devices like s3c6410 and i.MX31 can completely boot from SD Card, including getting their bootloader from there. On i.MX31 using this scheme, we are able to boot to a bash prompt in 2.5s from cold.

Another big problem with U-Boot was the inability to update the bootloader from the Linux world. Instead the bootloader was treated special and had to be updated over USB with DFU (the exclusivity of it enforced by an ECC policy disconnected from Linux, meaning U-Boot had to write it in there). In Qi, the bootloader can be updated by a packaged update like anything else in the rootfs.

There are no private environment / states, there are per-board heuristics for where to look for a kernel, and filesystem files which can append to kernel commandlines if present. Therefore identical devices with the same Qi patchlevel will always act the same.

-Andy


Tim Bird <tim.bird@...>
 

Matt Hsu wrote:
Summary: Refactoring Qi, lightweight bootloader.
I've created a page for this at:
http://elinux.org/CELF_Project_Proposal/Refactor_the_Qi_lightweight_bootloader

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================


Rob Landley
 

On Thursday 17 December 2009 02:31:36 Matt Hsu wrote:
Summary: Refactoring Qi, lightweight bootloader.

Proposer: Matt Hsu


Description:

Qi (named by Alan Cox on Openmoko kernel list) is a minimal bootloader that
"breathes life" into Linux. Its goal is to stay close to the minimum
needed
Which bits does it do?

Every piece of software needs something to initialize the DRAM controller.
After that, they could presumably just jump to a known location in flash that
the kernel lives at, and the kernel can decompress itself and so on. Doing
just that much can probably be done in 4k pretty easily. (I gloss over the
kernel command line and root filesystem location, which can be hardwired into
the image if you really don't care about providing the developer with a UI.)

However, if that's your minimum then you can't use the bootloader to re-flash
the device, which is kind of handy. (It gives you an un-bricking fallback
short of pulling out a jtag.) But doing that requires things like a network
driver, TCP/IP stack, tftp implementation, serial driver, command line
interpreter, and so on. And of course code to erase and write flash blocks for
your flash chip du jour, plus knowledge of the flash layout. (In theory, said
knowledge comes from parsing a device tree.)

I still live in hope that somebody will split the first part (coreboot) out
from the second part (grub) in embedded bootloaders. It's sad that the PC has
a tradition of orthogonality here but the embedded world treats it as a single
opaque lump.

http://wiki.openmoko.org/wiki/Qi
Looking at the screen shot there, you've got code to parse ext2 filesystems.
What is your definition of "minimal"?

Rationale for not providing a boot menu is you don't want to mess with video
init. I don't think I've actually seen an embedded bootloader that messes
with video, they do serial console instead, and you have a screen shot of
serial console messages so apparently the serial driver part is there...

Confused,

Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds


Matt Hsu <matt@...>
 

Rob Landley wrote:
However, if that's your minimum then you can't use the bootloader to re-flash the device, which is kind of handy. (It gives you an un-bricking fallback short of pulling out a jtag.)
Hi Rob,

Well, Boot from SD is your good friend.

If you look at the platform that Qi which is supported, most of them all have this feature.
If you notice the trend of SoC, booting from peripherals becomes a must.

Once you step into kernel via Qi, kernel provides you everything such as mtd utils to re-flash device.
We don't need to support programming the device in the bootloader anymore.
Don't reinvent the wheel.

Looking at the screen shot there, you've got code to parse ext2 filesystems. What is your definition of "minimal"?
Enough to boot into Linux.
Rationale for not providing a boot menu is you don't want to mess with video init.
Nope, the centric idea of Qi, is let kernel deal with everything it could handle.
The video init should be handled by kernel stead of bootloader.

The following clip demonstrate the advantage of Qi bootloader:

http://www.youtube.com/watch?v=ol9LWBKXXwQ&feature=related

- Faster booting time
- Get rid of flash on display device when stepping into kernel

Hope these could clear your doubt.

Cheers, Matt


Rob Landley
 

On Sunday 20 December 2009 23:51:23 Matt Hsu wrote:
Rob Landley wrote:
However, if that's your minimum then you can't use the bootloader to
re-flash the device, which is kind of handy. (It gives you an
un-bricking fallback short of pulling out a jtag.)
Hi Rob,

Well, Boot from SD is your good friend.
Ok, not aiming to be a generic bootloader then. You're only trying to support
hardware that has the flash equivalent of a floppy drive. Got it.

If you look at the platform that Qi which is supported, most of them
all have this feature.
Because if they didn't you wouldn't support them? Bit of a selection bias
there...

Life is a bit easier if you can stick a USB port on your device and boot from
a USB stick or cdrom. Most of the SoCs coming down the pipe support USB too,
but your wiki doesn't seem to consider this an interesting case...

If you notice the trend of SoC, booting from peripherals becomes a
must.
Depends how cheap you want your hardware to be, and whether you can afford
separate development and deployment boards.

Software development is a bit easier when you can spec adding a extra few
dollars worth of hardware to your device rather than redoing its software. I
tend to deal with people who repurpose existing cheap plastic crap already
being knocked out in huge quantities somewhere in china. (Their hardware
contribution _might_ be a different plastic case for the thing.) I've also
dealt with highly integrated little suckers that haven't got _space_ for an sd
card. Since I one day dream of following Moore's Law down to "disposable
computing", I'm reluctant to discard these cases as uninteresting.

Once you step into kernel via Qi, kernel provides you everything
such as mtd utils to re-flash device.
We don't need to support programming the device in the bootloader
anymore.
Depending on the kernel to reflash the device means that if you reflash the
device with a kernel that doesn't work, what you have is a brick. There's
lots and lots of reasons for a kernel not to work, and a 2.6 kernel kernel
takes up around a megabyte on a device that may only have 2 or 4 megs of ram
so keeping an old one around at all times isn't feasible. So without some
kind of fallback (such as a little ~32k bootloader at the start of flash that
you never overwrite, in its own little erase block), every time you install a
new kernel you risk bricking the device. (If you only care about devices that
have 2 gigs of flash, life is much easier.)

Don't reinvent the wheel.
There are how many existing bootloaders out there already?

Looking at the screen shot there, you've got code to parse ext2
filesystems. What is your definition of "minimal"?
Enough to boot into Linux.
You need to parse an ext2 filesystem to boot into linux? (I'm not saying it's
a _bad_ thing, I'm just not seeing how it's "minimal".)

Rationale for not providing a boot menu is you don't want to mess with
video init.
Nope, the centric idea of Qi, is let kernel deal with everything it
could handle.
So the fact the kernel can provide a serial console means you shouldn't?

The video init should be handled by kernel stead of bootloader.
Oh agreed. What that has to do with a command line interpreter on the serial
console was my question.

Personally, i'm used to embedded devices being headless. (I tend to consider
the iPhone the next stage in the mainframe->micro->mini evolution and it'll
soon be "just another PC" target when they get the ergonomics worked out.
Heck, give those suckers a USB port and you could give 'em a big keyboard and
display today, and they could even charge themselves through the thing. Way
more powerful than my first 386 PC. Actually about as powerful as the laptop I
had circa 2002 or 2003.)

The following clip demonstrate the advantage of Qi bootloader:

http://www.youtube.com/watch?v=ol9LWBKXXwQ&feature=related

- Faster booting time
I.E. you enable the cpu cache during kernel decompression.

I believe the u-boot guy just posted that as a todo item, and that falls on
the coreboot side of bootloading, which really should be broken out into its
own little project someday...

The other delay in something like u-boot is it pausing to see if it gets an
"esc" or similar from the serial console, so it can be interrupted and provide
a prompt. That's a configurable delay which can be removed if it really bugs
you.

- Get rid of flash on display device when stepping into kernel

Hope these could clear your doubt.
Your proposal confused me when it said things like "improve portability" and
"make people spend less time on bootloader development", so I thought you were
aiming at something more generic. But this bootloader is not intended to ever
apply to something like a hammer board, or any existing linksys-class router,
most current cell phones, devices like handheld game consoles where the sd
card stores data but not the operating system (due to the thing still needing
to work when you swap sd cards)...

My mistake. You're creating a bootloader specifically tailored to booting from
sd cards. Might want to make that clear on the web page.

Cheers,
Matt
Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds


Andy Green <andy@...>
 

On 12/21/09 08:00, Somebody in the thread at some point said:

Hi Rob -

However, if that's your minimum then you can't use the bootloader to
re-flash the device, which is kind of handy. (It gives you an
un-bricking fallback short of pulling out a jtag.)
There is a simple and much more flexible alternative to have a "recovery rootfs" on the device that can be selected, by holding down a button or somesuch which is understood by Qi. That gives you ALL the options possible in Linux like network connectivity for your recovery without doubling the support load in having a "smart" bootloader and its different drivers to deal with.

Well, Boot from SD is your good friend.
Ok, not aiming to be a generic bootloader then. You're only trying to support
hardware that has the flash equivalent of a floppy drive. Got it.
No, Qi will boot from onboard NAND or NOR as well as SD.

For other reasons -- eg, unbrickability -- true boot from SD is very powerful. By "true boot from SD" I mean the on-CPU ROM is pulling a bootloader from SD so the entire software environment is coming from there.

If you look at the platform that Qi which is supported, most of them
all have this feature.
Because if they didn't you wouldn't support them? Bit of a selection bias
there...
Naturally enough we only added support for devices we are using in the real world :-)

All the devices Qi target have what Samsung called "steppingstone", some small SRAM on the die that can hold your core bootloader startup from NAND or SD (since you cannot execute direct from NAND). That's probably a better one-liner to describe the scope of it.

Life is a bit easier if you can stick a USB port on your device and boot from
a USB stick or cdrom. Most of the SoCs coming down the pipe support USB too,
but your wiki doesn't seem to consider this an interesting case...
To be clear Openmoko Wiki isn't "our Wiki". Qi got started there to solve the difficulties created by having basically a second Linux on the device in the form of U-Boot. But the main work on Qi is done now at warmcat and Matt's OMAP branch. Openmoko doesn't seem to be in the phone business any more.

Depending on the kernel to reflash the device means that if you reflash the
device with a kernel that doesn't work, what you have is a brick. There's
That does not follow if you have a recovery kernel + initrd / rootfs in another partition that can be selected by holding down a key.

new kernel you risk bricking the device. (If you only care about devices that
have 2 gigs of flash, life is much easier.)

Don't reinvent the wheel.
There are how many existing bootloaders out there already?
So what?

As described the advantages of Qi's debloated and Linux-centric philosophy came from having our noses pushed for months into the world of crap coming from supporting multiple drivers and power behaviours spread over U-Boot and Linux.

We did not find any existing open ARM bootloader that had a solid internal structure and avoided the bloat.

Therefore we made one that proves that it is possible to push most of the features of U-Boot and other heavy bootloaders to Linux.

Looking at the screen shot there, you've got code to parse ext2
filesystems. What is your definition of "minimal"?
Yes the main tenet is it has got to "load" and "boot" the kernel. If the kernel is on ext* then that's part of its remit.

Enough to boot into Linux.
You need to parse an ext2 filesystem to boot into linux? (I'm not saying it's
a _bad_ thing, I'm just not seeing how it's "minimal".)
The ext* code is tight (taken from U-Boot and cleaned up).

With VFAT, ext*, memory test, all of the features Qi comes to 28KBytes typical binary.

Rationale for not providing a boot menu is you don't want to mess with
video init.
Nope, the centric idea of Qi, is let kernel deal with everything it
could handle.
So the fact the kernel can provide a serial console means you shouldn't?
Qi provides serial output for diagnosis of the boot action. But not input.

But the answer to your rhetorical question is "yes". It costs mindspace and support effort to support and work with these complex chunks of software, it's real money.

Turn your question around, what is your rationale for duplicating the stuff that Linux does great into a bloated bootloader with its own shell? If you look at each of the features in a bootloader like U-Boot that is not directly needed for "load" and "boot", they each have a more powerful counterpart in Linux already. You have to support the Linux version on your Linux box, just simplify it down to that alone.

The video init should be handled by kernel stead of bootloader.
Oh agreed. What that has to do with a command line interpreter on the serial
console was my question.
First maybe it's on you to explain why, other than habit, you think that is worth the bloat.

I.E. you enable the cpu cache during kernel decompression.
You are assuming wrongly that the kernel is compressed. On iMX31 I use Qi with uncompressed 3MByte kernel from SD Card, it's booted to shell from cold in 2.5s. In any event with compressed kernels, we use the kernel uncompress, it's not done in Qi.

aiming at something more generic. But this bootloader is not intended to ever
apply to something like a hammer board, or any existing linksys-class router,
most current cell phones, devices like handheld game consoles where the sd
card stores data but not the operating system (due to the thing still needing
to work when you swap sd cards)...

My mistake. You're creating a bootloader specifically tailored to booting from
sd cards. Might want to make that clear on the web page.
Apologies for throwing you back into confusion and doubt by the fact Qi boots fine from NAND, NOR and SD.

Actually Qi will work fine on most current ARM-based cellphones and game consoles with NAND or NOR.

-Andy


Wolfgang Denk
 

Dear Andy,

In message <4B29F834.90108@...> you wrote:

Another big problem with U-Boot was the inability to update the
bootloader from the Linux world. Instead the bootloader was treated
special and had to be updated over USB with DFU (the exclusivity of it
enforced by an ECC policy disconnected from Linux, meaning U-Boot had to
write it in there). In Qi, the bootloader can be updated by a packaged
update like anything else in the rootfs.
Can you explain to me why it was not possible to update U-Boot from
Linux? I cannot imagine a reason for such a restriction.

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@...
I have made mistakes, but have never made the mistake of claiming I
never made one. - James G. Bennet


Mike Frysinger <vapier.adi@...>
 

On Mon, Dec 21, 2009 at 14:30, Wolfgang Denk wrote:
Andy wrote:
Another big problem with U-Boot was the inability to update the
bootloader from the Linux world.  Instead the bootloader was treated
special and had to be updated over USB with DFU (the exclusivity of it
enforced by an ECC policy disconnected from Linux, meaning U-Boot had to
write it in there).  In Qi, the bootloader can be updated by a packaged
update like anything else in the rootfs.
Can you explain to me why it was not possible to update U-Boot from
Linux? I cannot imagine a reason for such a restriction.
we do it all the time under the Blackfin port on nor/nand/spi flashes.
-mike


Andy Green <andy@...>
 

On 12/21/09 19:30, Somebody in the thread at some point said:

Hi -

Let me add back the context for this quote:

''While working at Openmoko maintaining their kernel, it became obvious that U-Boot was turning into a mini-me for Linux. Many cut-down Linux drivers were appearing there, there was a shell with an environment holding private states ...''

Another big problem with U-Boot was the inability to update the
bootloader from the Linux world. Instead the bootloader was treated
special and had to be updated over USB with DFU (the exclusivity of it
enforced by an ECC policy disconnected from Linux, meaning U-Boot had to
write it in there). In Qi, the bootloader can be updated by a packaged
update like anything else in the rootfs.
Can you explain to me why it was not possible to update U-Boot from
Linux? I cannot imagine a reason for such a restriction.
I was talking about GTA02 specifically here, it was (and still is AFAIK) only updateable by DFU for the bootloader. We had kernels with soft ECC that differed from the ECC / bad block marking generated and used by the s3c2442 NAND hardware unit. If U-Boot wrote it, it could at least read it again. So DFU was / is the only official way to update GTA02 bootloader.

As you say nothing generally stops update of U-Boot from Linux same as anything else can be updated from there, if Linux can write those NAND sectors with ECC / BBT that the bootloader can understand or ignore, and Linux can understand and comply with / update the BBT scheme of U-Boot.

Another benefit of SD Card boot is that it regularizes the bootloader physical storage, so these issues don't exist and it can always be updated from Linux.

-Andy


Wolfgang Denk
 

Dear Andy Green,

In message <4B2FD7DB.10008@...> you wrote:

Another big problem with U-Boot was the inability to update the
bootloader from the Linux world. Instead the bootloader was treated
...
I was talking about GTA02 specifically here, it was (and still is AFAIK)
only updateable by DFU for the bootloader. We had kernels with soft ECC
that differed from the ECC / bad block marking generated and used by the
s3c2442 NAND hardware unit. If U-Boot wrote it, it could at least read
it again. So DFU was / is the only official way to update GTA02 bootloader.
This only means that the ports of U-Boot and Linux to that hardware
were inconsistent, which means that at least one of them can be
considered broken. [And I'm tempted to add that this might eventually
be a consequence of the fact that all this work was done without ever
getting in touch with the U-Boot community. Likewise, no code or
fixes have been contributed back into mainline.]


I think it's unfair to blame U-Boot for a poor on incorrect
implementation of the NAND driver on this specific hardware.


Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@...
It's hard to make a program foolproof because fools are so ingenious.


Andy Green <andy@...>
 

On 12/21/09 21:38, Somebody in the thread at some point said:

Hi -

I was talking about GTA02 specifically here, it was (and still is AFAIK)
only updateable by DFU for the bootloader. We had kernels with soft ECC
that differed from the ECC / bad block marking generated and used by the
s3c2442 NAND hardware unit. If U-Boot wrote it, it could at least read
it again. So DFU was / is the only official way to update GTA02 bootloader.
This only means that the ports of U-Boot and Linux to that hardware
were inconsistent, which means that at least one of them can be
Actually the problem of synchronizing the ECC and BBT rules between bootloader and Linux is a generic one where raw NAND is involved. Soft ECC and Hard ECC on a particular chip may differ and the kernel decides what policy it has there. It's a "raw NAND" issue not a U-Boot one but if anyone shipped a kernel one way and wanted to change it, they have to take care about matching the bootloader's view of the NAND in great detail if they want interoperation between Linux and the bootloader.

considered broken. [And I'm tempted to add that this might eventually
be a consequence of the fact that all this work was done without ever
getting in touch with the U-Boot community. Likewise, no code or
fixes have been contributed back into mainline.]
Well, this is getting a bit off-topic for considering the merits of Qi.

Openmoko started to use U-Boot on GTA01 long before my time, likewise the soft ECC was "like that when I got there".

About tapping into the wisdom of the U-Boot community, most of their changes were GTAxx-specific. For example I don't know any other Linux device than GTA02 with a Glamo in it, there is a lot of code I ported from Linux for that bloating their tree. With closed docs, that would be completely useless for upstream.

Bearing in mind they could only update by DFU and with GTA01, there was no bootloader recovery mechanism if it failed, keeping their bootloader tree inhouse was an established tradition by the time I got there.

For all those reasons the best way we found was deprecate their U-Boot tree and learn from what we had found there. (A fair amount of Qi is cut out of U-Boot so I see the great work it has done as well as the problems: respect for your work.)

I think it's unfair to blame U-Boot for a poor on incorrect
implementation of the NAND driver on this specific hardware.
Yeah it would be unfair to solely blame U-Boot for the sins of GTA02. But it's fair to blame U-Boot with the sins of U-Boot.

The main lessons I took from that was the dollar and time value of removing the "unnecessary features" in U-Boot and especially the Openmoko tree of it:

- video drivers

- shells

- environments

- special update mechanisms

- raw NAND at all

- duplicating the OS in there

- private nonvolatile state

- PMU management when we are already able to run

- per board variant bootloader image (ie, GTA02 v3 can only run a special GTA02 v3 binary of U-Boot that can't run on anything else; Qi has a per CPU binary that supports all variants)

After understanding that all those are needless and actively wasteful in developer resource, support, and device time the reasoning led to the development of Qi in its current form.

-Andy


Robert Schwebel
 

Andy,

On Mon, Dec 21, 2009 at 10:38:31PM +0000, Andy Green wrote:
About tapping into the wisdom of the U-Boot community, most of their
changes were GTAxx-specific. For example I don't know any other Linux
device than GTA02 with a Glamo in it, there is a lot of code I ported
from Linux for that bloating their tree. With closed docs, that would
be completely useless for upstream.
Hmm, there have been interesting items in the openmoko trees. For
barebox, we took the DFU support, which was done in a device specific
way, cleaned that up and made a generic command out of it:

dfu /dev/self0(bootloader)sr,/dev/nand0.root.bb(root)

You can specify the slots on the command line, not hardcoded. Whereas we
reworked the interfaces, the core code was pretty interesting. So I
think some items it would have been worth to be pushed into u-boot at
the time it was written.

Bearing in mind they could only update by DFU and with GTA01, there
was no bootloader recovery mechanism if it failed,
Our DFU scenario goes like "press a button while booting goes into DFU
mode", so you can re-flash as often as you like. However, our use cases
are probably different than yours (deeply embedded systems, which often
don't even have removable stuff like SD or USB sticks).

In general, I like in-system techniques much better than card juggeling,
because it fits better into automated environments like our RemoteLab,
which does our automatic nightly tests. But that's surely a matter of
the use case you have.

The main lessons I took from that was the dollar and time value of
removing the "unnecessary features" in U-Boot and especially the
Openmoko tree of it:
In barebox, we use Kconfig to configure things away; so removing
unnecessary features is just a matter of 'make menuconfig'.

- video drivers
I see video drivers in the bootloader as an optimization topic: If you
can effort to get your splash 3 s after power-on, you should leave video
drivers out of the boot loader and do it all in the kernel.

Our competition in industry projects is often the old 2-lines-alpha
displays, which are "instant on" after you hit the power switch. If this
is required, I don't see a way to achieve that with kernel-only at the
moment.

- shells
Especially during development, we often see that the hardware people
really like having a very limited shell with hardware bit banging access
in barebox. In a phase where you port Linux to a device, it gives you
something that works while Linux is not ready yet. And in barebox, you
have full scripting capabilities, so hardware people can even use that
for certain qualification scripts.

- environments
That was one of our design goals in barebox as well: get rid of the
scripting in the environment, as it was done in u-boot.

- special update mechanisms
What do you mean with "special"?

- raw NAND at all
- duplicating the OS in there
If you want to boot from NAND-only devices, how would you do that
without NAND drivers?

- private nonvolatile state
?

- PMU management when we are already able to run
Several CPUs need PMU support early in the boot stage, because they come
up in slow-clock mode. So you either boot slow until the kernel is up
far enough (but then the whole kernel loading is slow), or you need
access to the PMU from the bootloader.

In barebox, our design is that we have frameworks for i2c+spi to access
a PMU, but if you don't need that, you can configure it away. The idea
is that *if* you actually need it, then better have a good design for
it.

- per board variant bootloader image (ie, GTA02 v3 can only run a
special GTA02 v3 binary of U-Boot that can't run on anything else;
Qi has a per CPU binary that supports all variants)
I don't know the GTA02 hardware, but it is often a problem to actually
detect a certain CPU or board variant on runtime. But if that's
possible, I don't see a reason why you can't make a single image.

We had one strange case with an MPC5200 where the bootup bus
configuration was so complicated that it was a major issue and we
decided for different bootloaders, but in general it should be possible.

rsc
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |


Andy Green <andy@...>
 

On 12/21/09 23:19, Somebody in the thread at some point said:

Hi Robert -

Thanks for your reply.

mode", so you can re-flash as often as you like. However, our use cases
are probably different than yours (deeply embedded systems, which often
don't even have removable stuff like SD or USB sticks).
Right, some of what Qi proposes won't work on all systems, like SD boot where there is no SD card. But the core "just load and boot" heuristic should work almost anywhere.

- special update mechanisms
What do you mean with "special"?
Hmm, there have been interesting items in the openmoko trees. For
barebox, we took the DFU support, which was done in a device specific
way, cleaned that up and made a generic command out of it:
DFU is a "special update mechanism" which I believe is a bad idea.

I know a lot of people are still putting out full rootfs images as updates, and for some platforms that are too resource-constrained that's all people can do.

But for modern devices like ARM11+ and the kind of board they typically find themselves on with a network connection, these are fundamentally at the level of PC from a few years ago. Linux PCs then and now use packaged update systems to manage the software on the device. And they package both the kernel and the bootloader and track and update it like any other package, apply packagesets as transactions, etc. The correct approach I believe is to unify the bootloader (and kernel) update path with the rest of the system, all done from Linux alone.

(Personally I used Fedora ARM port and RPM, but any distro and packagesystem like Debian workable on ARM would be fine).

dfu /dev/self0(bootloader)sr,/dev/nand0.root.bb(root)

You can specify the slots on the command line, not hardcoded. Whereas we
reworked the interfaces, the core code was pretty interesting. So I
think some items it would have been worth to be pushed into u-boot at
the time it was written.

Bearing in mind they could only update by DFU and with GTA01, there
was no bootloader recovery mechanism if it failed,
Our DFU scenario goes like "press a button while booting goes into DFU
mode", so you can re-flash as often as you like. However, our use cases
are probably different than yours (deeply embedded systems, which often
don't even have removable stuff like SD or USB sticks).
The issue GTA01 faced was that you are updating the thing the button takes you to. If that goes south you have to bust out JTAG / OpenOCD and that is definitely not an end-user tool for a consumer product.

In GTA02 a separate NOR was added to contain the "bootloader behind the button" which was not updatable in the field, that then caused trouble since the updatable NAND bootloaders moved on but that never did. It also acted as the third pole in the love triangle betweeen NAND U-Boot and Linux in the NAND ECC / BBT differences since it could only recover the NAND bootloader only with the NOR bootloader's fixed idea of what ECC and BBT looked like, no matter what we had done with updates to the NAND bootloader in the meanwhile (eg, move from soft to incompatible but faster hard ECC in Linux). So we were actually unable to migrate to hard ECC in Linux, which is an insane outcome of a broken system.

In contrast if your chip supports it (iMX31 and s3c6410 do and Qi works with those) having your bootloader on some sectors of SD card is wonderfully simple and easy to dd in on a postinstall scriptlet of your bootloader package.

In general, I like in-system techniques much better than card juggeling,
because it fits better into automated environments like our RemoteLab,
which does our automatic nightly tests. But that's surely a matter of
the use case you have.
Agreed.

But consider this: if your bootloader is on SD, and your bootloader completely rejects to hold private state on the board (other than onetime individualization, eg MAC address), something awesome happens when you pop your SD card and put it in another board, it comes up like the previous board did, no ifs or buts.

You can imagine the effect that has on production / test "virgin" board bringup. When you have seen this, you do not want to return to raw onboard NAND.

The main lessons I took from that was the dollar and time value of
removing the "unnecessary features" in U-Boot and especially the
Openmoko tree of it:
In barebox, we use Kconfig to configure things away; so removing
unnecessary features is just a matter of 'make menuconfig'.
That is good, but what I am suggesting is that

- these things are definitively unnecessary, ie, they deserve permanent deselection

- the config system leads to bootloader-binary-per-variant Hell

Because Qi burns off all the peripheral support and leaves it to Linux, actually building in support for multiple boards and multiple variants is pretty lightweight. The CPU bringup is always the same, SDRAM bringup may vary slightly and kernel commandlines and paths, amount and maybe placement of memory will change.

Qi uses a per-board callback in an API struct to discover at runtime which supported board it's on, and the board can check version bits on GPIO typically to discover which variant it is (which is passed on to Linux in an ATAG).

- video drivers
I see video drivers in the bootloader as an optimization topic: If you
can effort to get your splash 3 s after power-on, you should leave video
drivers out of the boot loader and do it all in the kernel.

Our competition in industry projects is often the old 2-lines-alpha
displays, which are "instant on" after you hit the power switch. If this
is required, I don't see a way to achieve that with kernel-only at the
moment.
Yeah that is true. You are into a 1.8 - 2 second (on iMX31 SD boot) delay from hitting the button to your driver starting up in Linux and getting your display up.

Given what you get out of that from a project management POV, I don't think 2 seconds for startup feedback is a problem for most systems. If your system has a hardwired power LED, then even more so.

But if you have to have the display lit quicker, Qi has per-board API callback that lets the board set itself up how it needs. You could add this there if you have to.

Have a look at

http://git.warmcat.com/cgi-bin/cgit/qi/tree/src/cpu/imx31/txtr-steppingstone.c?h=txtr

scroll down to the bottom to see how the per-board setup works.

- shells
Especially during development, we often see that the hardware people
really like having a very limited shell with hardware bit banging access
in barebox. In a phase where you port Linux to a device, it gives you
something that works while Linux is not ready yet. And in barebox, you
have full scripting capabilities, so hardware people can even use that
for certain qualification scripts.
Yeah I agree hardware people like doing that. Here's how that innocent pastime can take you to Hell.

I described on the Openmoko list how even normally good programmers become "like a fat girl in Ibiza" when they see how it is in (Openmoko tree anyway) U-Boot, any wild thing goes. (It was quite sad to have to chop down some of the drivers that had pretty good code quality from Linux to fit the simplified world in U-Boot). And some people who describe themselves as "hardware guys" are not good programmers.

What it led to was private bootloader trees that did not track the main one, filled with perverted bit-twiddling code that was not understood by anyone except the guy who wrote it, and that guy left a while back as did the guy after him.

These trees were not even on the radar of the software guys nor did any patches come. But it is these decayed stump versions of the bootloader forked years ago that will become the basis of production test in a huge expensive factory "because it has the test code in it". By now it's test code nobody really understands (even if they are told The Secret of its existence) and they daren't uplevel their tree (even if they know such black magic is possible) because they neither have the forked version unchanged any more nor have heard of revision control outside the context of homework.

Because it was an unknown secret whispered only to new initiates in the Hardware Club, nobody in the software world is trying to keep compatibility with this forked bootloader with resulting car-crashes. And indeed a fourth pole in the NAND / ECC policy love quadrangle if we're still counting.

Same thing happens if you allow the existence of "test kernels" as with "test bootloaders".

Ultimately, even if that had all been correctly managed, it is still not preferable to have anything but truly core hardware tests in the bootloader (ie, testing of assets required to boot Linux that may not already be working since we are running the bootloader: just SDRAM test normally) compared to having them in Linux, since they can be scripted and reported easily from Linux.

Therefore the only test code in Qi is SDRAM test, no special bootloader version is needed (or allowed in my case) for verification or test.

If rapid asset verification is needed, it should be done in Linux with stub drivers or added to machine init code temporarily, and in revision control of someone who will write the real driver.

All other test actions should be integrated into the Linux driver and if they need to be triggered, exposed down /sys.

All of that should be present in normal shipping kernels, so what you take to the factory is simply current shipping version of bootloader and kernel with no custom build of anything.

- environments
That was one of our design goals in barebox as well: get rid of the
scripting in the environment, as it was done in u-boot.

- raw NAND at all
- duplicating the OS in there
If you want to boot from NAND-only devices, how would you do that
without NAND drivers?
If all you have is NAND on your board then nothing can be done.

But if you have NAND and SD, it is possible

- private nonvolatile state
?
Private nonvolatile state is stuff like the U-Boot environment that lives on the board itself and is out of any update management.

This leads to the situation where two boards from the same factory can act totally differently depending on what opaque different secrets have been hidden away in their private nonvolatile state, even if everything updatable in the rootfs is at the same patchlevel and even the bootloaders themselves at the same patchlevel.

That is "private nonvolatile state Hell".

- PMU management when we are already able to run
Several CPUs need PMU support early in the boot stage, because they come
up in slow-clock mode. So you either boot slow until the kernel is up
far enough (but then the whole kernel loading is slow), or you need
access to the PMU from the bootloader.
Yeah. But in the PMUs I have seen, Vcore is not by default at the level where it can ONLY run at 32kHz or whatever. Instead it is at some intermediate voltage like 1.2V by default that will allow midrange operation. (On this iMX31 board I currently work on in fact the PMU comes up by default on Vcore high enough for 532Mhz directly.)

That enables you to complete the boot at a reasonable speed without actually having the requirement to touch the PMU in those cases.

In barebox, our design is that we have frameworks for i2c+spi to access
a PMU, but if you don't need that, you can configure it away. The idea
is that *if* you actually need it, then better have a good design for
it.
Yeah Qi has generic gpio bitbang i2c implemented already and we can do the same for SPI if needed. But I think you find most PMU have Vcore by default at a place you can run at a reasonable speed without touching it.

- per board variant bootloader image (ie, GTA02 v3 can only run a
special GTA02 v3 binary of U-Boot that can't run on anything else;
Qi has a per CPU binary that supports all variants)
I don't know the GTA02 hardware, but it is often a problem to actually
detect a certain CPU or board variant on runtime. But if that's
possible, I don't see a reason why you can't make a single image.
Yeah if care wasn't taken to reserve some GPIO for the task, it can be nontrivial. But assets like NOR can be detected with a VID / PID and used for this to fingerprint a board.

-Andy


Robert Schwebel
 

Hi Andy,

[is this the right set of lists to discuss these issues? It's not
directly CELF related, but I don't know a better place for general
project independend bootloader discussions]

On Tue, Dec 22, 2009 at 08:22:27AM +0000, Andy Green wrote:
DFU is a "special update mechanism" which I believe is a bad idea.
For dedicated embedded system in the non-phone league, with small root
filesystems, it works pretty well.

I know a lot of people are still putting out full rootfs images as
updates, and for some platforms that are too resource-constrained
that's all people can do.
Ah, ok, we don't do that.

But for modern devices like ARM11+ and the kind of board they
typically find themselves on with a network connection, these are
fundamentally at the level of PC from a few years ago. Linux PCs then
and now use packaged update systems to manage the software on the
device. And they package both the kernel and the bootloader and track
and update it like any other package, apply packagesets as
transactions, etc. The correct approach I believe is to unify the
bootloader (and kernel) update path with the rest of the system, all
done from Linux alone.

(Personally I used Fedora ARM port and RPM, but any distro and
packagesystem like Debian workable on ARM would be fine).
Until now, we are using the "build it yourself" approach with ptxdist,
basically because of these reasons:

- If something goes wrong, we want to be able to fix it, which means
that we must be able to recompile everything. Having the source is no
value by itself, if you are not able to build it.

- Root filesystems are small; a complete rootfs for a typical industrial
application with Linux, glibc, dbus and qt is about 20...30 MiB.

- People don't change software that often, and if they do, it has to be
made sure that it is absolutely well tested. Nobody wants to reboot
their deeply embedded machine controller at the other end of the world
if somehting goes wrong. We usually don't have an administrator who
can interoperate if something goes wrong.

- Customizability. We recently tried Debian on the Neo, and it is an
absolute mess. About 2.5 minutes of boot time, a lot of flicker and
almost no reactivity of the system. So for us, the question remains
how to customize standard distributions in a reproducable way.

So at least at the moment, I prefer ptxdist over a customized debian.
But in general, I respect the argument why people want to use standard
distributions (I know the pain to fix all the cross compiling issues). I
just don't think that today's distributions are there yet. Most embedded
systems I've seen so far which follow the strip-down-standard-distro
pattern have been unreproducable for anyone but the original developer.

So we were actually unable to migrate to hard ECC in Linux, which is
an insane outcome of a broken system.

In contrast if your chip supports it (iMX31 and s3c6410 do and Qi
works with those) having your bootloader on some sectors of SD card is
wonderfully simple and easy to dd in on a postinstall scriptlet of
your bootloader package.
Agreed.

In general, I like in-system techniques much better than card
juggeling, because it fits better into automated environments like
our RemoteLab, which does our automatic nightly tests. But that's
surely a matter of the use case you have.
Agreed.

But consider this: if your bootloader is on SD, and your bootloader
completely rejects to hold private state on the board (other than
onetime individualization, eg MAC address), something awesome happens
when you pop your SD card and put it in another board, it comes up
like the previous board did, no ifs or buts.

You can imagine the effect that has on production / test "virgin"
board bringup. When you have seen this, you do not want to return to
raw onboard NAND.
In general, I agree with you here (although I think the MAC address
should be glued to the hardware and not change if you change SD cards ->
people will then copy it and you have the same MAC address twice).

However, I think it's more developer friendly to have that "no changable
state" as a policy than a design decision: during development, we quite
often change for example the kernel command line (adding quiet or debug
switches, boot from net/disk...). For delivery, we just make barebox +
it's scripting environment one image and change that to r/o, if
necessary. So you can get best of both worlds.

In barebox, we use Kconfig to configure things away; so removing
unnecessary features is just a matter of 'make menuconfig'.
That is good, but what I am suggesting is that

- these things are definitively unnecessary, ie, they deserve
permanent deselection

- the config system leads to bootloader-binary-per-variant Hell
For us, the bootloader is not only something which is delivered with the
product - that's one use case. But there is also quite a long time where
lots of developers work with the board - and in that use case we'd like
to do things like hacking on registers (without the complexity of
Linux), do TFTP/nfsroot, change kernel command lines etc.

Seeing the production case, I'm all with you.

Qi uses a per-board callback in an API struct to discover at runtime
which supported board it's on, and the board can check version bits on
GPIO typically to discover which variant it is (which is passed on to
Linux in an ATAG).
Unfortunately, not all hardware vendors make different variants
detectable in software. That's quite often a problem for us. So for the
general case, a compile time selection is necessary. If the hardware
behaves, you are right :-)

I see video drivers in the bootloader as an optimization topic: If you
can effort to get your splash 3 s after power-on, you should leave video
drivers out of the boot loader and do it all in the kernel.

Our competition in industry projects is often the old 2-lines-alpha
displays, which are "instant on" after you hit the power switch. If this
is required, I don't see a way to achieve that with kernel-only at the
moment.
Yeah that is true. You are into a 1.8 - 2 second (on iMX31 SD boot)
delay from hitting the button to your driver starting up in Linux and
getting your display up.
... and we still do have a lot of ARM9 systems in the 200...400 MHz
range out there.

I described on the Openmoko list how even normally good programmers
become "like a fat girl in Ibiza" when they see how it is in (Openmoko
tree anyway) U-Boot, any wild thing goes.
That's why we went the device model way in barebox. Having a restricted
environment is no excuse for hackery (and people even assumed that the
same binary size wouldn't be possible in the beginning).

And some people who describe themselves as "hardware guys" are not
good programmers.
Very true ;)

What it led to was private bootloader trees that did not track the
main one, filled with perverted bit-twiddling code that was not
understood by anyone except the guy who wrote it, and that guy left a
while back as did the guy after him.
That's solvable by working on mainline integration. You'll get this
problem with Linux as well, if people are not on a mainline strategy. No
tool can change that.

All other test actions should be integrated into the Linux driver and
if they need to be triggered, exposed down /sys.
Ack.

If all you have is NAND on your board then nothing can be done.

But if you have NAND and SD, it is possible
In barebox, we have bootloader images that run from everywhere. So you
can for example write a little script that detects that you run from SD
or USB stick (taken that we'll have drivers for them) and relocate
to somewhere else (NAND on ARM, or soldered SSD on x86).

- private nonvolatile state
Private nonvolatile state is stuff like the U-Boot environment that
lives on the board itself and is out of any update management.
On modular systems (like phyCORE, Qseven etc) you have a CPU module, a
baseboard, maybe additional addon boards, and requirements where to
store information like MAC addresses, serial numbers etc. are often very
different.

This leads to the situation where two boards from the same factory can
act totally differently depending on what opaque different secrets
have been hidden away in their private nonvolatile state, even if
everything updatable in the rootfs is at the same patchlevel and even
the bootloaders themselves at the same patchlevel.
You can make the private nonvolatile state r/o or w/once.

That enables you to complete the boot at a reasonable speed without
actually having the requirement to touch the PMU in those cases.
Unfortunately, often people want to boot as fast as possible, which
requires optimization in that area as well. We recently had a board
which refused to boot without the PMIC having switched on some voltages
which are default-off.

Yeah Qi has generic gpio bitbang i2c implemented already and we can do
the same for SPI if needed. But I think you find most PMU have Vcore
by default at a place you can run at a reasonable speed without
touching it.
My fear is that quite often one starts with "oh, this problem is simple,
let's design simple". Then things move on and you notice that you need
to work on SPI, I2C, you need ext2, jffs2, ubifs, later maybe btrfs,
then SD support or USB. In the end, the problem turns out to be
complicated.

That's why we more or less stayed with the overall look-and-feel of
u-boot in barebox. We just tried to pull in the best ideas from the
Linux and POSIX universe, like the device model, sane scripting etc.
That way, people at least have something where they can put their hacks
if they really need them, without too much damage for the rest of us :-)

rsc
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |


Andy Green <andy@...>
 

On 12/22/09 11:12, Somebody in the thread at some point said:

Hi Robert -

(Personally I used Fedora ARM port and RPM, but any distro and
packagesystem like Debian workable on ARM would be fine).
Until now, we are using the "build it yourself" approach with ptxdist,
basically because of these reasons:

- If something goes wrong, we want to be able to fix it, which means
that we must be able to recompile everything. Having the source is no
value by itself, if you are not able to build it.
Fedora provides a whole solution there, with the restriction it's designed for native build, not cross. That puts a limit on what you can target with it, which is why "ARM11+". You can just go here

http://fedoraproject.org/wiki/Architectures/ARM#Getting_Started_with_the_Fedora_ARM_Port

Get a rootfs tarball, bring it up and build packages with the compiler and build stuff in there already, yum install other stuff, etc. Again SD rootfs makes that simple to consider and do, you can untar it from your PC them replug to your device and run it.

- Root filesystems are small; a complete rootfs for a typical industrial
application with Linux, glibc, dbus and qt is about 20...30 MiB.
True, typical Fedora footprint for X and so on will be 200MB.

However. The developer cost in using these distro packages is next to nothing, both acquiring them and their ongoing uplevelling when fc13 comes. Quality of the packages is better than I can make :-) and it's pretty much anything there is to want

http://ftp.linux.org.uk/pub/linux/arm/fedora/pub/fedora/linux/releases/12/Everything/arm/os/Packages/

And again, if you have SD Card rootfs basis, smallest one of those now you can get in a shop is 512MB anyway.

- People don't change software that often, and if they do, it has to be
made sure that it is absolutely well tested. Nobody wants to reboot
their deeply embedded machine controller at the other end of the world
if somehting goes wrong. We usually don't have an administrator who
can interoperate if something goes wrong.
Packages only help any QA effort. You don't have to release every package build to your stable repo, a staged development / testing / stable repo scheme is simple.

- Customizability. We recently tried Debian on the Neo, and it is an
absolute mess. About 2.5 minutes of boot time, a lot of flicker and
:-| Yeah. But I admired the effort they put in there. The CPU is too weak for that game though and the Glamo in there is the final insult. (Actually there is a good rage comic: "Problem, GTA02?")

almost no reactivity of the system. So for us, the question remains
how to customize standard distributions in a reproducable way.
Well, I can explain what I did with Fedora, we were up in Bash with ethernet usb gadget and sshd ready for action in < 5s from cold on iMX31 SD Card boot. That's the stock Fedora bash, sshd, etc from their tarball.

The main action is not to use the stock /sbin/init. (You can still use sysV init / upstart to bring services up and down even if init never got started since it's basically all sat there in /etc/init.d.)

So at least at the moment, I prefer ptxdist over a customized debian.
But in general, I respect the argument why people want to use standard
distributions (I know the pain to fix all the cross compiling issues). I
just don't think that today's distributions are there yet. Most embedded
systems I've seen so far which follow the strip-down-standard-distro
pattern have been unreproducable for anyone but the original developer.
Right. Fedora is different though, there are no cross-built packages (although they do provide cross compilers, I use them for kernel and Qi builds) and if storage is sufficient, there's no need to strip anything out. Just nobble init.

But consider this: if your bootloader is on SD, and your bootloader
completely rejects to hold private state on the board (other than
onetime individualization, eg MAC address), something awesome happens
In general, I agree with you here (although I think the MAC address
should be glued to the hardware and not change if you change SD cards ->
people will then copy it and you have the same MAC address twice).
(That's what I meant by "other than onetime individualization eg MAC address" -- that stuff (only) should live on the board not the card)

However, I think it's more developer friendly to have that "no changable
state" as a policy than a design decision: during development, we quite
often change for example the kernel command line (adding quiet or debug
switches, boot from net/disk...). For delivery, we just make barebox +
it's scripting environment one image and change that to r/o, if
necessary. So you can get best of both worlds.
Right. Qi supports this by allowing the boot source filesystem that holds the kernel to also contain an optional "append" text file. The text in this file, if present, is appended to the generated kernel commandline. Qi auto generates the correct root= (based on where it found the first workable kernel) and other board-mandated options already, but this external append file lets you do things like force debuglevel=8 from the filesystem without doing anything to the bootloader itself.

It's a plain text file, public, exposed in a standard filesystem and will get copied along with the rootfs. So it's flexibility without the private nonvolatile bootloader storage problems.

Unfortunately, not all hardware vendors make different variants
detectable in software. That's quite often a problem for us. So for the
general case, a compile time selection is necessary. If the hardware
behaves, you are right :-)
Well for my uses of it I have been able to specify that we should have versioning GPIOs on the boards. It's a good idea anyway.

... and we still do have a lot of ARM9 systems in the 200...400 MHz
range out there.
Same as GTA02 class processor. Qi loads the kernel there in a few seconds and brings up its video in Linux as told.

What it led to was private bootloader trees that did not track the
main one, filled with perverted bit-twiddling code that was not
understood by anyone except the guy who wrote it, and that guy left a
while back as did the guy after him.
That's solvable by working on mainline integration. You'll get this
problem with Linux as well, if people are not on a mainline strategy. No
tool can change that.
It's nothing to do with mainline, just intra-company communication and management failure.

But if you have NAND and SD, it is possible
In barebox, we have bootloader images that run from everywhere. So you
can for example write a little script that detects that you run from SD
or USB stick (taken that we'll have drivers for them) and relocate
to somewhere else (NAND on ARM, or soldered SSD on x86).
Qi works the same. The same image can be placed in NAND, NOR or SD and boot any supported device.

Private nonvolatile state is stuff like the U-Boot environment that
lives on the board itself and is out of any update management.
On modular systems (like phyCORE, Qseven etc) you have a CPU module, a
baseboard, maybe additional addon boards, and requirements where to
store information like MAC addresses, serial numbers etc. are often very
different.
What is the point there?

This leads to the situation where two boards from the same factory can
act totally differently depending on what opaque different secrets
have been hidden away in their private nonvolatile state, even if
everything updatable in the rootfs is at the same patchlevel and even
the bootloaders themselves at the same patchlevel.
You can make the private nonvolatile state r/o or w/once.
You can do that with individualization data, like MAC Addresses. But that is not what the U-Boot environment is.

That enables you to complete the boot at a reasonable speed without
actually having the requirement to touch the PMU in those cases.
Unfortunately, often people want to boot as fast as possible, which
requires optimization in that area as well. We recently had a board
which refused to boot without the PMIC having switched on some voltages
which are default-off.
If your device is able to run from USB power, there's the issue that you are limited to 100mA before enumeration takes place. So without a USB stack, you have to trade speed for power anyway.

Yeah Qi has generic gpio bitbang i2c implemented already and we can do
the same for SPI if needed. But I think you find most PMU have Vcore
by default at a place you can run at a reasonable speed without
touching it.
My fear is that quite often one starts with "oh, this problem is simple,
let's design simple". Then things move on and you notice that you need
to work on SPI, I2C, you need ext2, jffs2, ubifs, later maybe btrfs,
then SD support or USB. In the end, the problem turns out to be
complicated.
You're right to fear it because you are too willing to re-introduce the bloat into your bootloader. For example you mention earlier that "Unfortunately, often people want to boot as fast as possible" and that is the rationale for re-introducing PMU management and the serial bus driver back into the bootloader. But actually, normal customers don't care about 200ms on boot either way. They can get the thing to market quicker and so cheaper and more reliably without that stuff in the bootloader.

That's why we more or less stayed with the overall look-and-feel of
u-boot in barebox. We just tried to pull in the best ideas from the
Linux and POSIX universe, like the device model, sane scripting etc.
That way, people at least have something where they can put their hacks
if they really need them, without too much damage for the rest of us :-)
No offence but you are basically describing U-Boot there, mini Linux, scripting, hacks. Surely like U-Boot dreams of growing up into Linux, you will find your project dreaming of growing up into U-Boot.

Qi rejects all of those and so is different (and smaller) than your project, I am sure there is "room in the market" for all of the philosophies including U-Boot.

-Andy


Robert Schwebel
 

Hi Andi,

On Tue, Dec 22, 2009 at 10:23:37PM +0000, Andy Green wrote:
Fedora provides a whole solution there, with the restriction it's
designed for native build, not cross.
That's probably also a matter of taste. I still find it a feature to be
able to cross compile the systems - we can still recompile whole systems
consistently in just a few (ten) minutes, i.e. in order to change a
central switch (like softfloat vs. hardfloat, use another toolchain
etc). Out of interest, how long does it take to recompile Fedora on for
example MX31?

Packages only help any QA effort. You don't have to release every
package build to your stable repo, a staged development / testing /
stable repo scheme is simple.
For let's say a telematics box it's almost impossible to test packet
combinations. All you can do is decide for "update application" and
"update platform". If you got your software updates via SMS, it
shouldn't go wrong because of an untested packet combination :-)

Right. Fedora is different though, there are no cross-built packages
(although they do provide cross compilers, I use them for kernel and
Qi builds) and if storage is sufficient, there's no need to strip
anything out. Just nobble init.
Hmm, I'm still not convinced. But as I don't have any data, I'll keep
quiet :-)

I think a central question is if you want to optimize things like

* early boot splash
* footprint

Well for my uses of it I have been able to specify that we should have
versioning GPIOs on the boards. It's a good idea anyway.
Hardware description is still one of the unsolved problems out there. It
works good if you have just a few variants. We often have for example
5-10 assembly options on a cpu module, plus several on the base board,
all not in-system detectable. We tried several variants to describe the
hardware in some config block, we tried to have oftree sniplets in the
hardware, but none of the options did really work well. In the end, you
need the schematics in machine readable form... When oftree comes into
the ARM world, we need a maximum-style bootloader which can provide
oftrees.

What it led to was private bootloader trees that did not track the
main one, filled with perverted bit-twiddling code that was not
understood by anyone except the guy who wrote it, and that guy
left a while back as did the guy after him.
That's solvable by working on mainline integration. You'll get this
problem with Linux as well, if people are not on a mainline
strategy. No tool can change that.
It's nothing to do with mainline, just intra-company communication and
management failure.
My experience is that mainline exposure of that perverted bit-twiddling
code and ideas helps finding sane solutions, because someone will
allways ask the right questions :-)

Unfortunately, often people want to boot as fast as possible, which
requires optimization in that area as well. We recently had a board
which refused to boot without the PMIC having switched on some
voltages which are default-off.
If your device is able to run from USB power, there's the issue that
you are limited to 100mA before enumeration takes place. So without a
USB stack, you have to trade speed for power anyway.
My argument is that there are all these different requirements out
there: your use cases, mine, others. What we want to do with barebox is
turning u-boot into something that can fulfill all of these
requirements, not only parts of it. You don't need a network driver in
your bootloader? Just configure it out and don't look at the code. But
somebody else may care, and he has a sane design to put his driver into.
The maximum paradigm works as good as the minimum does.

My fear is that quite often one starts with "oh, this problem is
simple, let's design simple". Then things move on and you notice
that you need to work on SPI, I2C, you need ext2, jffs2, ubifs,
later maybe btrfs, then SD support or USB. In the end, the problem
turns out to be complicated.
You're right to fear it because you are too willing to re-introduce
the bloat into your bootloader. For example you mention earlier that
"Unfortunately, often people want to boot as fast as possible" and
that is the rationale for re-introducing PMU management and the serial
bus driver back into the bootloader. But actually, normal customers
don't care about 200ms on boot either way. They can get the thing to
market quicker and so cheaper and more reliably without that stuff in
the bootloader.
That's a matter of the definition of "normal customers" :-)

No offence but you are basically describing U-Boot there, mini Linux,
scripting, hacks. Surely like U-Boot dreams of growing up into Linux,
you will find your project dreaming of growing up into U-Boot.
Well, we think we have found a good design for reducing the hack factor
significantly.

While I'm still convinced that, for our projects, we have to stay with
barebox in maximum-mode, ptxdist's cross-building of self-made
distributions for now, there is a gradual move towards mainstream
distributions, where people can just 'apt-get install kde'. All our
recent experiments seem to suggest that mainstream distros are not able
to do what we need today, but times may change.

My conclusion is that we are on a good track: ptxdist is a 'do things in
a reproducable way, following a configuration' style system which is not
limited to cross building by design. I can imagine that, once the new
world has arrived, we can also use it for customizing standard
distributions as well. And barebox is flexible enough to serve the
maximum-bootloader pattern today, while gradually migrate towards the
minimum one.

However, I found the discussion pretty interesting! Thanks a lot, it
gave me some interesting insights.

rsc
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |


Andy Green <andy@...>
 

On 12/22/09 23:28, Somebody in the thread at some point said:

Hi -

Fedora provides a whole solution there, with the restriction it's
designed for native build, not cross.
That's probably also a matter of taste. I still find it a feature to be
The two ways have different advantages and disadvantages. Cross gives you speed of rebuild, but 99% of packages on an ARM11+ type system you actually have no interest in rebuilding since you just use the stock versions. Cross can be extremely painful with projects that are not set up for it, eg, perl. Native build gives you compatibility of build with the whole package universe. The slow build action doesn't hurt so much because -->

able to cross compile the systems - we can still recompile whole systems
consistently in just a few (ten) minutes, i.e. in order to change a
central switch (like softfloat vs. hardfloat, use another toolchain
etc). Out of interest, how long does it take to recompile Fedora on for
example MX31?
I don't know or care when I get the binary packages from a repo where they're already built. The whole point of a distro solution is someone did all the work for you. You're only thinking about mass rebuild yourself because it's the buildroot mindset, that whole task disappears with a distro basis.

Packages only help any QA effort. You don't have to release every
package build to your stable repo, a staged development / testing /
stable repo scheme is simple.
For let's say a telematics box it's almost impossible to test packet
combinations. All you can do is decide for "update application" and
"update platform". If you got your software updates via SMS, it
shouldn't go wrong because of an untested packet combination :-)
You missed my point I think. You can emulate "issue 6 monthly rootfs tarball updates" by just updating the stable package repo at long intervals with well-tested packagesets. At the same time you can offer other repos with newer features earlier, get changed packages tested easier, confirm patchlevel on test systems, etc.

Right. Fedora is different though, there are no cross-built packages
(although they do provide cross compilers, I use them for kernel and
Qi builds) and if storage is sufficient, there's no need to strip
anything out. Just nobble init.
Hmm, I'm still not convinced. But as I don't have any data, I'll keep
quiet :-)
You can get data easily enough, use the Fedora tarball and run with init=/bin/bash as a starting point. Then write a script that does the minimal jobs like bringup lo, remount rw, config ethernet and service sshd start and time that as init.

The point is you're definitively bypassing the *whole* of Fedora bringup that way while still getting the advantages of the Fedora rootfs basis and most of sysv service management.

I think a central question is if you want to optimize things like

* early boot splash
* footprint
Early boot splash is a kernel matter, it has facilities for it already. For footprint, you can't play this game if you don't have say 200MB spare as already told it's the cost of entry.

But if you can play the distro game (combined with leaving the bootloader alone to only boot Linux) the software engineering and maintainence load for a really capable rootfs dwindles to about the same level of job as maintaining a Linux server box, and quality goes up.

Well for my uses of it I have been able to specify that we should have
versioning GPIOs on the boards. It's a good idea anyway.
Hardware description is still one of the unsolved problems out there. It
...
need the schematics in machine readable form... When oftree comes into
the ARM world, we need a maximum-style bootloader which can provide
oftrees.
I take your point but actually there's no reason the *bootloader* needs that when the bootloader is focussed solely on booting Linux. *Linux* might want an equipment list from the board, but then typically you would build all the drivers and they can simply probe and fail if its not there on the board.

It's nothing to do with mainline, just intra-company communication and
management failure.
My experience is that mainline exposure of that perverted bit-twiddling
code and ideas helps finding sane solutions, because someone will
allways ask the right questions :-)
I'm not sure I managed to give the flavour of a bunch of hardware guys half a world away rotating in and out on Military service. Even patches internally aren't happening, Mainline isn't an answer.

Unfortunately, often people want to boot as fast as possible, which
requires optimization in that area as well. We recently had a board
which refused to boot without the PMIC having switched on some
voltages which are default-off.
If your device is able to run from USB power, there's the issue that
you are limited to 100mA before enumeration takes place. So without a
USB stack, you have to trade speed for power anyway.
My argument is that there are all these different requirements out
there: your use cases, mine, others. What we want to do with barebox is
Sure, Qi targets ARM9+ with steppingstone. If you have something else you will have to use something else or port it.

If it's not understood what the advantages are of Qi's strict rejection of its list of evil things, then they will likely go with U-Boot I guess since it has the mindshare.

bus driver back into the bootloader. But actually, normal customers
don't care about 200ms on boot either way. They can get the thing to
market quicker and so cheaper and more reliably without that stuff in
the bootloader.
That's a matter of the definition of "normal customers" :-)
What I mean by it is for geeks like us, it's interesting to see how fast it will go. The actual customer cannot tell 200ms by eye he will accept it if it's not passing his threshold of being "too slow". But he will like getting it shipping earlier because the bootloader is almost invisible in dev effort and in management of production.

distributions, where people can just 'apt-get install kde'. All our
recent experiments seem to suggest that mainstream distros are not able
to do what we need today, but times may change.
Well it sounds like you didn't try Fedora which should be high on the list. But if "what you need" doesn't match I guess you won't have a good time there no matter how great it is.

However, I found the discussion pretty interesting! Thanks a lot, it
gave me some interesting insights.
You're welcome, we even mentioned Qi occasionally :-)

-Andy


Andy Green <andy@...>
 

On 12/23/09 02:28, Somebody in the thread at some point said:

Hi -

No TCP/IP, no TFTP, not even BOOTP (but it's a nice bonus), no command
line interpreter (just a GPIO on board to boot into "unbrick me" mode
:-), and most strikingly _no_ flash driver for flash chip du jour.

To flash it you send a kernel to boot from RAM which is capable of
flashing it.
Sorry I missed where this kernel appears from and the bootloader that spawned it, since both could get trashed. That is actually a conundrum on a lot of systems and some of the solutions (write-once backup bootloader) in the long run lead to other issues.

True SD Boot does truly deliver unbrickability if you are willing to swap out or reformat the SD card.

http://wiki.openmoko.org/wiki/Qi
Looking at the screen shot there, you've got code to parse ext2 filesystems.
What is your definition of "minimal"?
Ew, ext2 doesn't even satisfy powerfail-during-kernel-upgrade safety.
It's just misleading (but accurate). ext2 is the "lowest common denominator" read-only parsing that actually supports ext3 and ext4 if you are careful about the formatting options. So the actual filesystem is ext3 or ext4 typically (ext3 in GTA02 case), it's not that the bootloader is mandating specifically ext2.

I agree it does beg the question of what is "minimal".

The proposal did explain quite well what Qi aims for: not duplicating
lots of kernel drivers badly. If it succeeds in the area of flash
writing, network drivers, network protocols and so on it would be no
bad thing.
Thanks.

One area for potential common ground among bootloaders could be to
share the code for parsing filesystems. It'd be great to see that in
a library shared by GRUB, Qi, U-boot and so on as it's not device
specific at all and not particularly large, but not so trivial that
it's good to have lots of clones.
Yeah it's not a bad idea.

It's possible to boot without parsing filesystems, but that is one
rather nice feature, and with the right filesystems it can make system
updates powerfail-safe.
Bootloader is tricky, but actually on this iMX31 device Fedora is used, yum update keeps the last 3 kernels around and our kernel package follows that. So it's possible to have backup kernels automatically integrated into the bootloader and packaging system.

Rationale for not providing a boot menu is you don't want to mess with video
init. I don't think I've actually seen an embedded bootloader that messes
with video, they do serial console instead, and you have a screen shot of
serial console messages so apparently the serial driver part is there...
In perspective, serial is usually quite simple. Output only serial is
even simpler, though :-)
Totally agree!

-Andy


Robert Schwebel
 

On Wed, Dec 23, 2009 at 08:38:08AM +0000, Andy Green wrote:
I don't know or care when I get the binary packages from a repo where
they're already built. The whole point of a distro solution is someone
did all the work for you. You're only thinking about mass rebuild
yourself because it's the buildroot mindset, that whole task
disappears with a distro basis.
If you don't step into for example toolchain problems or other crazy
things...

You can emulate "issue 6 monthly rootfs tarball updates" by just
updating the stable package repo at long intervals with well-tested
packagesets. At the same time you can offer other repos with newer
features earlier, get changed packages tested easier, confirm
patchlevel on test systems, etc.
Yes, that's a valuable option.

I take your point but actually there's no reason the *bootloader*
needs that when the bootloader is focussed solely on booting Linux.
*Linux* might want an equipment list from the board, but then
typically you would build all the drivers and they can simply probe
and fail if its not there on the board.
The oftree is currently provided by the bootloader, and much of what it
contains is unprobable peripherals, i.e. the IP cores in the SoC cpus.
For example for i.MX (which we happen to maintain in the mainline),
there is a strong aim for having one kernel that runs on as many devices
as possible. If you want to do this and if you can't probe significant
parts of the hardware, you need an instance outside of the kernel who
tells you what's actually there.

I'm not sure I managed to give the flavour of a bunch of hardware guys
half a world away rotating in and out on Military service. Even
patches internally aren't happening, Mainline isn't an answer.
Well, there are many projects out there which are not so secret that one
cannot expose the kernel drivers. And even if they are, it is possible
to establish a peer-review culture inside of a corporation. But you
won't get the full power of community review. That's the trade off one
has to accept for having secrets :-) But quality is generally a big
issue.

bus driver back into the bootloader. But actually, normal
customers don't care about 200ms on boot either way. They can get
the thing to market quicker and so cheaper and more reliably
without that stuff in the bootloader.
That's a matter of the definition of "normal customers" :-)
What I mean by it is for geeks like us, it's interesting to see how
fast it will go. The actual customer cannot tell 200ms by eye he will
accept it if it's not passing his threshold of being "too slow". But
he will like getting it shipping earlier because the bootloader is
almost invisible in dev effort and in management of production.
We have customers who care about "splash in 0.5 s" vs. "shell runs after
3 s, then qt starts". People may be used to that kind of noticable boot
time in the phone business, but in the industry (where embedded Linux
boxes are even more "devices" than computers) they often are not.

Do you remember the times when we had analog TV? We could zap through 5
channels in under 3 seconds. *That's* performance :-) My sattelite
receiver needs about 10 seconds to boot. Sometimes it feels like
innovation goes backwards.

Cheers,
rsc
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |