WimTool, Some help needed - no programmer skills needed! External links Options

[booty#1]

Hi community,

as some of you already know I am working on replacement for ImageX that can be freely distributed (in difference to ImageX which can be legally only obtained as part of the WAIK).

First I made good progress in understanding the WIM file format but now I am confronted with some problems regarding the used compression algorithms:

1. The compression algorithm used for fast compression is currently unknown. I documented all the details I know about it in WimTool Wiki.

Now I want to find out which algorithm is really used. My algorithm comparison method is rather simple but usually effective:

- compress LGPL.txt using a compression tool containing the algorithm to be tested
- open the output file on a hex editor
- search if there is a large binary block that is identical to the compressed file of the WIM file (the compressed LGPL.txt in test_fast.wim starts at offset 0xD0)

A large number of compression tools provided by Microsoft has to be tested and that is the part I am hoping to get some support by all of you here on boot-land. All you need can be found in this attachment:  wimtest.zip ( 39.36K ) Number of downloads: 296
.
Please perform the test outlined above and post the results. That would really help me knowing what algorithm it is respectively not is.

In the meanwhile I want to focus myself on finding a library that implements compression/decompression for the MS-LZX compression algorithm (used for compression level maximum).

I am counting on your help,
booty#1

[was_jaclaz]

Maybe a "stupid" remark, but it seems like 7-zip:
1) fails to open test_none.wim
2) opens allright test_maximum.wim (LZX)
3) fails to open test_fast.wim

Something that may be unrelated, but knowing how MS guys like to re-use code, it may be not:
http://www.msfn.org/board/W98-Slip-genuine...ind-t81068.html
(Search for "Diamond" on the thread)

I will try the "usual suspects", and let you know.

Things to checK:
http://www.speakeasy.org/~russotto/chm/

http://www.cabextract.org.uk/libmspack/

(this latter one should support more formats)

jaclaz

[booty#1]

Maybe a "stupid" remark, but it seems like 7-zip:
1) fails to open test_none.wim
2) opens allright test_maximum.wim (LZX)
3) fails to open test_fast.wim

That is not stupid, that is simply a sad fact. 7zip's WIM implementation bases on reverse engineering and it only covers the maximum compression level (LZX algorithm), nothing else.

I will try the "usual suspects", and let you know.

Thanks, I really appreciate your help

Things to checK:
http://www.speakeasy.org/~russotto/chm/
http://www.cabextract.org.uk/libmspack/
(this latter one should support more formats)

The problem is that both are libraries not read-to-use tools. The first one is designed to be compiled under Linux (makes it not easy to port to Win32), the other one I already compiled on Win32/Visual Studio, but using it is really complicated. No easy to use interface at least not for the compression algorithm, only for the base file formats (CHM, CAB, ...). And documentation of the compression back end isn't very good.

booty#1

[was_jaclaz]

Found a reference here and on some other places

All WIM images feature one of two compression technologies - LZX or XPress. LZX gives greater compression when space restrictions are an issue, and is the option of choice when installing from fast media like a DVD.

Looking for "Xpress" compression, now.

If it is "another name" for lznt1 or lznt1+, dd for windows /FAU version) should support it:
http://gmgsystemsinc.com/fau/

I'll check and report.

jaclaz

[was_jaclaz]

This:
http://www.planet-source-code.com/vb/scrip...1&lngWId=-1

could be useful (I cannot get most of it, but you should be able to)

jaclaz

[was_jaclaz]

The dd for windows (FAU edition) lznt1 and lznt1+ -comp option appear to produce files almost, but not quite, completely unlike the original test_fast.wim

I have found some reference to the effect that lznt1 compression is the algorithm used in NTFS filesystem compression (which makes sense, since thhe .wim is, more or less, a filesystem).

This article explains how to use that kind of compression for Inno Setup scripts:
http://www.vincenzo.net/isxkb/index.php?ti...p;printable=yes

there may be something of worth in it.

jaclaz

PS: just found a somewhat related article (in Russian):
http://www.nf-team.org/drmad/zf/zf5/zf5_025.htm
I am attaching a google translation of it.

The output format of the dd for windows appears like the one described in the article, maybe there are DIFFERENT lznt1 algorithms.

Attached File(s)

 RussianPage.txt ( 9.35K ) Number of downloads: 150

 

[booty#1]

The dd for windows (FAU edition) lznt1 and lznt1+ -comp option appear to produce files almost, but not quite, completely unlike the original test_fast.wim

Can you express the alikeness in percent?

PS: just found a somewhat related article (in Russian):
http://www.nf-team.org/drmad/zf/zf5/zf5_025.htm
I am attaching a google translation of it.
The output format of the dd for windows appears like the one described in the article, maybe there are DIFFERENT lznt1 algorithms.

Thanks, the article was quite informative.

I found a reference to a small compression util that uses (according to the documentation) "XPRESS" as well:
BinCompress from the "Platform Builder for Microsoft Windows CE 5.0". From what I have read some C-header files and source code (partial) should be included as well. This platform builder package may be very helpful but it is not freely available. I only found an evaluation edition that fails on installation.

booty#1

[was_jaclaz]

likeness: 0%

On further tests, the dd for windows lznt1 appears (say 70%) similar to NTFS compression on disk.

So, it seems like the lznt1 algorithm is used (if at all) in a deeply modified way in XPRESS (Wim image "fast") compression.

jaclaz

P.S.: bincompress is available also in the 6.0 version:
http://msdn.microsoft.com/en-us/library/aa924411.aspx
See if this one installs:
http://www.microsoft.com/downloads/details...;displaylang=en

[was_jaclaz]

It seems that there was some activity on the matter when Longhorn came out, and before Ximage was made available, some header info that may be (or nay not) be useful:
http://board.iexbeta.com/index.php?showtopic=51516

jaclaz

[was_jaclaz]

@booty

Something more that may be of use:
http://gpwiki.org/forums/viewtopic.php?t=2623
http://gpwiki.org/forums/viewtopic.php?t=3037

there is a link to this compression Library:
http://www.pscode.com/vb/scripts/ShowCode....78&lngWId=3

though it seems to me it's related to the same topic of the Alex Ionescu's article I previously linked to:
http://www.planet-source-code.com/vb/scrip...1&lngWId=-1

More info from Alex:
http://www.alex-ionescu.com/Native.Pdf

I'll go on "stand-by", waiting for your checks on the above, will simply try some more archivers/analyzers/decompressors to see if anything works with test_fast.wim.

Would it be possible that "native" compression has changed in Vista?

Thus:
2K/XP "Native" compression = lznt1
Vista "Native" conpression=Xpress



jaclaz