Proposer: Frieder Ferlemann


Summary:
Upgrade an existing Hex editor to be able to read/write
Intel Hex and Motorola S-record files


Description:
These formats can have an important role in an embedded developers
toolchain. The formats specify both the addresses and the
corresponding data, thus allowing to specify specific ranges
of data which are to be modified. While editors for binary
files exist for linux they all seem to lack import and export
functionality for these formats.

Reading two files consecutively should result in a hexadecimal
display like:

00000000 02 00 08 80 fe 75 81 07 12 00 78 e5 82 ... ..u....x..
00000010 60 03 02 00 03 79 00 e9 44 00 60 1b 7a 00 90 00 `....y..D.`.z...

where the corresponding bytes are either highlighted (when
the definition in the 1st file differs from the 2nd file),
normal (if same), obmitted or greyed out (if there is no
definition in either of the files), be in color1 if only
defined in the 1st file and color2 if only defined in the 2nd.

Writing a selection of the buffer (lets assume byte 0x000b to 0x000f
in the example above) to an .hex or .s19 file should be possible
starting with offset 0x000b, 0x0000 or a user specified address.


Benefit:
If you own a mainboard/dvd-player/mouse/sound card/keyboard/
usb-device/graphics card/digital camera/microwave oven/mobile phone/
cdrom/car/pocket calculator/tv/modem/watch/printer/remote control/
washing machine it's likely that these file formats were involved.

Text tools (srecord, binutils) for these formats exist but might
present an inhibition for developers which are used to a graphical
toolchain on other platforms.

Embedded developers targetting these devices (especially 8-bit targets)
typically do not use Linux as a host platform. If there is a win over
then I'd hope for better linux support (f.e. upgrading the firmware
of some of these devices would not require buying the license of
another operating system) and I'd hope for more openness with respect
to protocols and linux drivers.

An anecdotical contribution relating to the importance of having
developers for a specific platform: http://www.youtube.com/watch?v=8To-6VIJZRE


Outlook:
In the long run it might be possible to interface to flashprogrammers.


Related work:
* okteta http://utils.kde.org/projects/okteta/
* srecord http://srecord.sf.net
* others http://en.wikipedia.org/wiki/Comparison_of_hex_editors
(* flashrom http://www.coreboot.org/Flashrom)


Related request:
A five year old request for khexedit (predecessor of okteta), 20 votes
http://bugs.kde.org/show_bug.cgi?id=94361


Persons to contact:
Friedrich W. H. Kossebau (okteta)
Peter Miller (SRecord)


Scope:
2 weeks+ (depending on the internal structure of the hex editor)

Dear Frieder Ferlemann,

In message <4B21251A.5060200@...> you wrote:

Summary:
Upgrade an existing Hex editor to be able to read/write
Intel Hex and Motorola S-record files


Description:
These formats can have an important role in an embedded developers
toolchain. The formats specify both the addresses and the
corresponding data, thus allowing to specify specific ranges
of data which are to be modified. While editors for binary
files exist for linux they all seem to lack import and export
functionality for these formats.

Hm... is this really needed? Ignoring the often existing restrictions
like segment structure, alignment to flash erase blocks etc. in an
image file (which render such image editing useless and can be dealt
with easier by regenerating the files through a normal build
process), typical image conversions / manipulations can be performed
often quite easily by combinations of standard tools like objcopy and
the like...

I doubt that a general purpose tool can be designed that covers all
potential use cases, and combinations of the already existing tools
are usually pretty efficient.

Just my $ 0.02...

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@...
God grant me the senility to accept the things I cannot change, The
frustration to try to change things I cannot affect, and the wisdom
to tell the difference.

On Thu, Dec 10, 2009 at 14:14, Wolfgang Denk wrote:

Frieder Ferlemann wrote:

Summary:
Upgrade an existing Hex editor to be able to read/write
Intel Hex and Motorola S-record files

Description:
These formats can have an important role in an embedded developers
toolchain. The formats specify both the addresses and the
corresponding data, thus allowing to specify specific ranges
of data which are to be modified. While editors for binary
files exist for linux they all seem to lack import and export
functionality for these formats.

Hm... is this really needed? Ignoring the often existing restrictions
like segment structure, alignment to flash erase blocks  etc.  in  an
image  file (which render such image editing useless and can be dealt
with  easier  by  regenerating  the  files  through  a  normal  build
process),  typical image conversions / manipulations can be performed
often quite easily by combinations of standard tools like objcopy and
the like...

I doubt that a general purpose tool can be designed that  covers  all
potential  use  cases, and combinations of the already existing tools
are usually pretty efficient.

indeed. if you want something to just edit files, how about:
objcopy -I ihex -O binary "$1" "$1.tmp" && khexedit "$1.tmp" &&
objcopy -I binary -O ihex "$1.tmp" "$1" && rm -f "$1.tmp"
-mike

Greetings Wolfgang and Mike,

Wolfgang Denk schrieb:

Summary:
Upgrade an existing Hex editor to be able to read/write
Intel Hex and Motorola S-record files


Description:
These formats can have an important role in an embedded developers
toolchain. The formats specify both the addresses and the
corresponding data, thus allowing to specify specific ranges
of data which are to be modified. While editors for binary
files exist for linux they all seem to lack import and export
functionality for these formats.

Hm... is this really needed?

Short answer: Yes, these formats were implemented for a reason.

Long answer falls into 3 parts which basically say: a) others seem
to need it, b) others implemented it, c) plausible usage scenarios
exist

Ad a) there were some votes for the bugs.kde.org request

Ad b) rethorical question: should the authors of
(http://www.hexedit.com/ http://www.kibria.de/frhed.html
http://www.dataman.com/ http://www.conitec.net/ ....)
be informed that they should drop intel hex/motorola s19
support as that feature is a non-issue?

Ad c) Take an 8-bit device with an 1k bootloader, a serial number,
a program and calibration data crammed into 8k
(or 16k/32k/64k if you are lucky - evidently this is not
a linux target:)
How do you to communicate the following usage cases:
1) upgrade only the program, 2) upgrade calibration data
but not the program, 3) upgrade bootloader, program and
calibration data but do not the touch serial number,
4) upgrade bootloader and program but do not the touch
serial number and do not touch calibration data.
This *can* be the matter of transmitting a *single* .hex or .s19
file respectively!
(you can do with binary files and some binutils/srecord scripts
too but you'd have to talk the customer into installing
some tools in a Windows environment first, possibly get around
limitations in command line length in above mentioned environment
and then try to educate the customer to run a batch file from
some directory where some files should reside and then feed
the resulting file somewhere else. For one or the other reason
one probably wouldn't do this for long:)
Or consider a binary file with the content of the nonvolatile
CMOS memory on a PC in which you'd want someone else (or a
group of people) to set two bytes at two different positions
to a specific value: You can tell to load the binary file into
the editor, then load the .hex/.s19 file over it, then save
again as binary.

At least being able to view and compare these files visually would be
helpfull. (Yes I know objcopy (I did mention binutils in the proposal),
srecord, hexdump and diff/tkdiff/kdiff3/xxdiff/kompare/winmerge))

Mike's workaround:

objcopy -I ihex -O binary "$1" "$1.tmp" && khexedit "$1.tmp" &&
objcopy -I binary -O ihex "$1.tmp" "$1" && rm -f "$1.tmp

is not there to fill the gap though. As it stands it rather seems to
illustrate the problem and argue in my direction.
And it does not yet handle an offset and additionally won't be able to
handle non contiguous .hex/.s19 files.

Ignoring the often existing restrictions
like segment structure, alignment to flash erase blocks etc. in an
image file (which render such image editing useless and can be dealt
with easier by regenerating the files through a normal build
process), typical image conversions / manipulations can be performed
often quite easily by combinations of standard tools like objcopy and
the like...

I doubt that a general purpose tool can be designed that covers all
potential use cases, and combinations of the already existing tools
are usually pretty efficient.

Yes, I sincerely doubt that a general purpose tool can be designed that
covers all potential use cases too. I never asked for anything close to
this though and I certainly would not use it if it existed.


My point is: there are Intel hex files and Motorola s-record files (and
probably half a dozen other formats which couple data with address
information).
These formats were invented, implemented (and are used) to tackle real
world tasks.

While there are tools to generate/manipulate them on the console
graphical tools to at least view them are lacking.
Beefing up an existing hex editor seemed reasonable to me.

Greetings,
Frieder

On Friday 11 December 2009 03:58:11 Frieder Ferlemann wrote:

Ad c) Take an 8-bit device with an 1k bootloader, a serial number,
a program and calibration data crammed into 8k
(or 16k/32k/64k if you are lucky - evidently this is not
a linux target:)

Linux may not run on it, but being able to control/program cheap plastic
embedded devices from a Linux host might be interesting too. (Allowing Linux
to be part of an embedded manufacturing process may be nice, even if the
result is a little 8-bit thing. Not sure what CELF's purview is.)

As price goes down, volume goes up. Being able to reflash ubiquitous little
disposable things from Linux is a lot better than having their manufacturing
process be some proprietary windows-centric thing.

Mike's workaround:

objcopy -I ihex -O binary "$1" "$1.tmp" && khexedit "$1.tmp" &&
objcopy -I binary -O ihex "$1.tmp" "$1" && rm -f "$1.tmp

is not there to fill the gap though. As it stands it rather seems to
illustrate the problem and argue in my direction.
And it does not yet handle an offset and additionally won't be able to
handle non contiguous .hex/.s19 files.

I plead the 5th.

I doubt that a general purpose tool can be designed that covers all
potential use cases, and combinations of the already existing tools
are usually pretty efficient.

Yes, I sincerely doubt that a general purpose tool can be designed that
covers all potential use cases too. I never asked for anything close to
this though and I certainly would not use it if it existed.


My point is: there are Intel hex files and Motorola s-record files (and
probably half a dozen other formats which couple data with address
information).

Sounds like a generic "binary diff" format. Does it allow insertion and
deletion of chunks, or just descriptions of changed byte ranges?

These formats were invented, implemented (and are used) to tackle real
world tasks.

Not all diffs are text. :)

While there are tools to generate/manipulate them on the console
graphical tools to at least view them are lacking.
Beefing up an existing hex editor seemed reasonable to me.

Which hex editor did you have in mind? (The first serious program I ever wrote
was a hex editor for the commodore 64. Ah, memories. That would have been...
1983? If so, I was 11.)

Rob
--
Latency is more important than throughput. It's that simple. - Linus Torvalds