[Celinux-dev] embedded Linux community - some ideas...

Tim Bird tim.bird at am.sony.com
Fri Sep 28 16:12:35 UTC 2012

Hi everyone,

I have for some time been considering the state of the embedded
Linux community, and some of the challenges facing us.  As an
embedded Linux developer myself, I am interested in improving
both my own working conditions and my efficiency and
effectiveness in my own projects.  But also I think we could
be having more fun, and learning and growing more if our
community were stronger.

In this message I am writing some of my own ideas, and would like
to solicit feedback on these and get other people's ideas as well.

I think the greatest challenge of developing embedded products
using open source software is fragmentation.  Fragmentation is
not new to open source, or software in general.  I have been
called out for saying that the issues that embedded contributors
have, relative to mainlining code, are different from those of
other developers.  That is, I've asserted that embedded is somehow
"special", and people (like Greg KH) have disagreed.

I think every group developing Linux has to identify those areas
of commonality that they can work on together, and separate them
from the areas of divergence, where there is not enough common ground
that it makes sense to collaborate or even attempt to push things
upstream.  In this regard (to disagree with Greg), I think embedded
*is* different than other Linux use areas. I think the very nature of
embedded, which at it's core consists of making customized solutions,
tends to run counter to the nature of open source, which tries to
leverage commonality and sharing.

And so I worry sometimes about how to increase our commonality,
and in particular how to encourage more sharing and discussion
within our community.

As a concrete example, I have been working lately on 4k
stacks for the ARM kernel.  The particular use case for this is
very special within Sony -- or at least it appears to me to be a
rather uncommon usage scenario.  But here's the thing...
How do I know there's not some other developer somewhere struggling
with something similar?  - maybe not the identical problem, but one
where 4K stacks would help their situation?

I attempted last year to submit my ARM 4K patches upstream.  However,
I got some pushback, and I didn't really see others with similar use
cases that required this, and so I didn't push it as hard as needed to
get it mainlined.  There will always be some weird things that really
shouldn't go upstream (and should be kept in someone's local
repository -- something that if they decide is needed can be
maintained out-of-tree.)  However, I've been bothered by the idea
that possibly I just didn't connect with the right people or find
those who might also benefit from this work.

And this led me back to thinking about the community.  One of the
foundational premises of open source is that of using network effects
to overcome shared challenges.  Eric Raymond's famous "to enough eyes,
all bugs are shallow" comes to mind.  If one can find enough people
with the same interests, open source offers a way to collaborate on
solutions rather than create and maintain them all by oneself.

I'm worried that even after years of trying to build community ties,
through things such as conferences, summits, the elinux wiki,
this mailing list, and the CE Linux Forum (and it's successor the
CE Workgroup), the embedded Linux developers still haven't
"come together" as a community as well as they could.

To be honest, I think that many of our previous efforts, while good
in their own right, have not caught fire the way we had hoped.  For
example, elinux.org has some great areas dedicated to BeagleBoard and
Raspberry Pi, and the section on boot time has lots of good material.
But the content in some other areas is outdated or otherwise lacking.
We have had occasional bursts of energy at improving the site
(sometimes spurred by a contest or conference event), but overall
we have not gotten a sustained population of contributors to the site.

eLinux.org is just one example.  There are numerous other examples of
places where there should be vibrant communication of embedded Linux
information and discussion of issues.  Among the best things we have, I
believe, is our conferences, (ELC, ELCE, the Japan Jamborees, and
various other events).  I'm always excited to meet with other developers
and hear about what they're working on.  But only a small number of
developers, out of the literally tens of thousands of embedded Linux
developers worldwide, can attend those.  And even then we only have
room for 40 to 50 presentation topics - which are, themselves, pretty
specialized.  We try to spread the information, through things
like publishing the presentations, and videos.  But, as worthwhile
as those are, it doesn't increase the interactivity of the community.

So, I have a few ideas I'd like to float out there:

1) I'd like to resurrect this list, and encourage people to use it
to discuss general topics of interest for embedded Linux.  I will be
trying over the next few months to discuss things related to system
size on this list, and in particular the projects I'm working on to
reduce kernel size.  I don't want this list to become LKML, where
the traffic is so high that everyone ignores it.  Hopefully we can
increase the traffic, but still keep it limited and relevant for those
who are working with general embedded Linux problems (you know, the
classics: size, boot-time, file systems, real-time, security,
graphics, embedded distros, etc.)

2) I'd like to try to get more focused groups at upcoming conferences.
Specifically, at ELC Europe and ELC I'd like to try to steer people
with common interests toward each other.  I'm not sure the exact
mechanism to use for this.  Other sub-communities tend to use
mini-summits for this.  The problem with these is that they are
limiting in terms of number of participants, and you kind of have to
know ahead of time who to invite.  I'd like to use a more open approach.
Sometimes just identifying the people who attended a particular
presentation will help you find those who are interested in or
actively working on a particular topic area.  The "hallway tracks"
of conferences are often where the most interesting and fruitful
discussions occur.  Somehow I'd like to introduce people of like
interests, and let them initiate their hallway tracks on their
own, without a lot of pre-invitations or formal setup.
Does anyone have ideas on how to do this?

To be less abstract: I'm leaning towards having a "size" summit
at ELC.  What's the best way to get people there who are (still)
interested in reducing the kernel and user-space size for embedded
Linux systems?

3) I'm looking for ways to invigorate the elinux.org wiki site.
I have a few projects that I'll (hopefully) be announcing in the
next few weeks (and, also hopefully, making a major push for at
ELCE in about 6 weeks).

Finally, 4) I'm also looking for ways to increase the involvement
in and use of the LTSI kernel.  I have a sense that it is starting
to get visibility and usage, but I have no solid indicators that
tell me yet whether a community is building or not around that
kernel version.

I'll start working on these by making a concerted effort to invite
people to the celinux-dev mailing list.  For historical reasons,
it doesn't have the most relevant name for what I'd like to achieve.
But it's a place to start (and, thankfully, it's administered by
someone besides me!).  Also, I'll try to post more often on this
list myself, to start to encourage the type of communication I'd like
to see more of in the future.

Am I the only one who feels like this?
(If so, it may be a frustrating couple of months, with a feeling like
I'm pushing on a string.)  Do you have ideas for how to build and
improve the embedded Linux community?  Or, do you have ideas for
what might be blocking it?

I've got lots more to say, but this is already long enough.
(Just wait until you hear my philosophy of kernel size reduction... ;-)

Thanks for reading this far...  Let me know what you think.
 -- Tim

P.S. Please encourage your embedded Linux developer buddies to subscribe
to this list!!  http://lists.celinuxforum.org/mailman/listinfo/celinux-dev

Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation
Senior Staff Engineer, Sony Network Entertainment

More information about the Celinux-dev mailing list