Digg this topic Add to my del.icio.us Submit to SlashDot 3 Pages V  < 1 2 3  
Reply to this topicStart new topic
> IMDISK by Olof Lagerkvist, Beyond FILEDISK and VDK
Rating 5 V
jaclaz
post May 11 2007, 08:56 AM
Post #21


Finder
***

Group: Advanced user
Posts: 1,622
Joined: 14-July 06
Member No.: 2


Italy


VERY good! (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)

What I had in mind were just a few small checks:
1) check for 55AA as last two bytes of first sector (that should mean that the drive image ALREADY has either a MBR or bootsector
2) check the first, say, three or four bytes of first sector and compare them to a "database" actually a simple .ini type of file will be enough (easily expandable/updatable) containing the start code of "known" MBR's and bootsectors and those for Qemu, vdk and similar vurtual disk images
3) consequently, display a message, something like:
a. In case a MBR is recognized, a further check for partition entries and:
QUOTE
This image appears to have a xxx OS MBR, with following partition entries:
1) Type 06 - BIGDOS Start at 0-1-1 Size 2040192 sectors
2) EPBR
E1) Type 0b - FAT 32 Start at 127-1-1 Size 8401932 sectors
3) - No entry
4) - No entry
Please choose which partition to mount......

b. In case a bootrecord is identified display nothing and mount the image
c. In case some kind of VM image is recognized display something like:
QUOTE
This appears to be a VM disktype image.
Are you sure you want to mount the image starting from first sector?
(Y/N) No will mount the image starting from offset xxx.

and the program would loop to 3a. with the new offset
d. In case code is not identified, display something like:
QUOTE
The image appears to contain uknown code in the first sector.
Are you sure you want to mount it starting from first sector?
Press ENTER for yes or input the number of sectors to skip

4) of course a couple added parameters for command line is necessary, something like
QUOTE
-f or -force

to skip the check and a
QUOTE
-o numsectors or -offset numsectors

to force skipping to a given value

As always, I have no idea on the amount of work is needed to realize something like the above, and as said the thing I am really missing is just the offset possibility.

Maybe one could first thing implement the offset thingy, and only later, when time permits, expand it to the full detection routine, for the benefit of users with less knowledge of disk image internals.

Also, I guess it would be possible to do it as an "external" program, being it a "real" .exe or a batch file, if you think this is the correct way, I can volunteer for the batch.

Thanks a lot for the "devio workaround" I can't wait to find the time to test it! (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)

(IMG:http://www.boot-land.net/forums/style_emoticons/default/cheers.gif)

jaclaz
Go to the top of the page
 
+Quote Post
Alexei
post May 11 2007, 03:01 PM
Post #22


Advanced Member
***

Group: .script developer
Posts: 647
Joined: 30-August 06
Member No.: 283



QUOTE (Olof Lagerkvist @ May 10 2007, 01:11 PM) *
Yes this is possible. The driver can be setup to auto-load at system start and all settings for auto-creating virtual disks can be set in the registry. There are no documentation for this so far. I will try to write something down soon to describe how this works.
This works as it is right now, and it does not need to be Windows in the server end, it could be a computer started on a Live-CD with FreeBSD. One problem though is that such drives cannot be mounted in the early stages of the OS boot. The problem is that for redirecting I/O over network the driver needs to communicate with a user-mode helper service and that one cannot be started early in the boot process.


Documentation: I found that not much explanations are needed if you provide a lot of examples, starting with trivial sample and adding features one by one. It also helps in testing, as it's easy to play with a sample writing down what's happening (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
Also, you have a lot of useful comments - just mark them somehow, then automatically extract to the docs.

I think, it will be possible to boot from the network (PXE server). Eventually, I'll take care of that (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
I understand why it's not possible, but I think I know how to do it (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)

I propose a feature (for better compatibility):
For each image create not only \\.\PHYSICALDRIVEx (drive), but also \\.\PHYSICALDRIVEn (volume),
where x-letter; n-number. The drive should have "fake MBR" with only one partition, which is the volume (IMG:http://www.boot-land.net/forums/style_emoticons/default/wink.gif)
All sectors from fake MBR till Boot Record are also supposed to be virtual (all 00h by default). MBR code, NT-signature, and (optionally) sectors are supposed to be user-definable.

Another idea is to add more control over data buffering:
- optional asynchronous load of a full image (in background)
- optional memory buffer with user-definable buffer size, policy (MRU?),
and percent of buffer which is non-pageable
- "fast mode" - return to caller right after writing to the buffer (temporary/dicardable data).
- "EWF mode" - changes are not reflected in source image
submodes: changes in memory, changes on another drive (local or virtual)

Other features:
- all command line options available from registry (just as one string?)
- Complete uninstall, as if it never been installed.
- user-definable policy on priorities of execution threads (useful?)

I took a quick look at your code. I can read "C", but not write (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
That's what I noticed (I may be wrong, though),
- IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS - not supported
- IOCTL_DISK_GET_DRIVE_GEOMETRY (etc.) does not return Media Type and Total Cyl
BTW, general expression: very good professional work (IMG:http://www.boot-land.net/forums/style_emoticons/default/thumbup.gif)

Regarding server-side of the communication proxy: I'd be interested in complete specification (I'm thinking about implementing it on windows). Even better it would be to have an API to a user-written DLL that provides virtual I/O (server side).
I mean something like this:
virtual disk<-->proxy client<-->transport<-->proxy server<--(API)-->UserDLL<-->whatever

(IMG:http://www.boot-land.net/forums/style_emoticons/default/cheers.gif)
Alexei

PS
@jaclaz,
In general, I dislike "automated logic" and numerous defaults. They may save 5min. of user time or make him (IMG:http://www.boot-land.net/forums/style_emoticons/default/frusty.gif) for a day (IMG:http://www.boot-land.net/forums/style_emoticons/default/wink.gif) An option to disable all automation should be a nice compromize (IMG:http://www.boot-land.net/forums/style_emoticons/default/wink.gif)

(IMG:http://www.boot-land.net/forums/style_emoticons/default/cheers.gif)
Alexei
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post May 11 2007, 04:38 PM
Post #23


Advanced Member
***

Group: Developer
Posts: 122
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


(This post has been modified a bit since I first posted it.)

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Documentation: I found that not much explanations are needed if you provide a lot of examples, starting with trivial sample and adding features one by one. It also helps in testing, as it's easy to play with a sample writing down what's happening (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)


All driver settings are in a very "techincal" format in the registry, basically based on the data structure the user mode applications send to the driver when a new virtual disk is created. You can create a registry key HKLM\SYSTEM\CurrentControlSet\Services\ImDisk\Parameters. Under that key the following values can be created.

MaxDevices
Optional value, default value is currently 16. If defined this should be a 32-bit (REG_DWORD) value and it can be used to specify the maximum number of virtual devices that can be created. This can be at most 32 because the driver keeps track of used device numbers in a 32-bit bit-field.

LoadDevices
This should be a 32-bit (REG_DWORD) value and it can be used to automatically create virtual disks when the driver is loaded. This means that if the driver is setup to load at system boot, virtual disks will be created at system boot. The value specifies the number of virtual disks to create in this way.

FileNameN
These values (string values of type REG_SZ) specifies which image file should be used for the virtual disks specified with the LoadDevices value. N should be replaced with 0 for the first virtual disk, 1 for the second and so on. Just like otherwise the filename is optional for RAM-disks and if a filename is specified for a RAM-disk this just specifies an image file to pre-load into the RAM-disk when it is created.

SizeN
These values should be 64-bit binary values and specifies the size for virtual disks. For file backed disks, the file size is adjusted to this size. N should be replaced with 0 for the first virtual disk, 1 for the second and so on. These parameters are optional for file backed virtual disks. Note that the Size values are specified in reverse byte order, the first byte represent the least significant byte of the 64-bit size.

FlagsN
These values specifies different options for creating the virtual disks. It should be a 32-bit (REG_DWORD) value. The flags are any reasonable combination of the flags specified in the source package in inc\imdisk.h (scroll down to "Bit constants for the Flags field in IMDISK_CREATE_DATA"). Note that most flags can be auto-selected so you will not always need to create the Flags value. For example, if you specify a size and no filename a r/w RAM-disk is created, if you specify a filename then a file backed virtual disk is created and so on.

As it is right now there is no error reporting whatsoever if any of the disks specified by LoadDevices cannot be created, neither to attached kernel debuggers nor to the system event log. I will add some error reporting in the future.

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Also, you have a lot of useful comments - just mark them somehow, then automatically extract to the docs.


Yes, I have been thinking about generating docs from comments with Doxygen but it did not understand Windows driver code very well so... Do anyone know anything better?

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
I think, it will be possible to boot from the network (PXE server). Eventually, I'll take care of that (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
I understand why it's not possible, but I think I know how to do it (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)


Good. I like such comments. (IMG:http://www.boot-land.net/forums/style_emoticons/default/cool.gif)

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
I propose a feature (for better compatibility):
For each image create not only \\.\PHYSICALDRIVEx (drive), but also \\.\PHYSICALDRIVEn (volume),
where x-letter; n-number. The drive should have "fake MBR" with only one partition, which is the volume (IMG:http://www.boot-land.net/forums/style_emoticons/default/wink.gif)
All sectors from fake MBR till Boot Record are also supposed to be virtual (all 00h by default). MBR code, NT-signature, and (optionally) sectors are supposed to be user-definable.


I can see that this could be useful but it requires lots of modifications both to the driver logic itself and to the way it registers the new virtual disks on the system so this will probably not be implemented in a near future.

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Another idea is to add more control over data buffering:
- optional asynchronous load of a full image (in background)


This should not be very difficult to implement. The image file is read into the memory by the newly created worker thread, but currently the calling thread waits for this to finish. I suppose there would be no big issues with just modifying this a little bit so that the image read-in is delayed to after the point the caller is waiting for.

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Other features:
- optional memory buffer with user-definable buffer size, policy (MRU?),
and percent of buffer which is non-pageable
- "fast mode" - return to caller right after writing to the buffer (temporary/dicardable data).
- "EWF mode" - changes are not reflected in source image
submodes: changes in memory, changes on another drive (local or virtual)


Interesting ideas. I have thought about some kind of write-copy mode that stores changed parts of the image in a memory buffer or in some kind of "undo file". The reason that is a heavy task to implement is that it needs some kind of cluster-indexing thingy in the background, some place where the driver can look in a table and see wether to read requested data from the original image file or from another place. But it is definitely something that I will try to implement in the future.

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Other features:
- all command line options available from registry (just as one string?)
- Complete uninstall, as if it never been installed.
- user-definable policy on priorities of execution threads (useful?)


All command line options are available, not as one string, but as described above.

The current uninstall routines should leave no traces. Is there anything in particular you think of here?

User-defined worker thread priority would probably be useful, yes, and hopefully not very difficult to implement as each virtual disk creates a worker thread in the System process and it can set which priority it likes when it is started.

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Other features:
I took a quick look at your code. I can read "C", but not write (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
That's what I noticed (I may be wrong, though),
- IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS - not supported
- IOCTL_DISK_GET_DRIVE_GEOMETRY (etc.) does not return Media Type and Total Cyl


IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS should only be implemented by disk drivers for volumes on hard disk drives and there are no hard disk drives associated with the ImDisk "partitions".

Which media type is returned depends on the drive type. It is for example 12 for virtual hard disk partitions and 11 for virtual CD/DVD-ROM drives. Total number of cylinders should be returned but possibly rounded down to nearest integer.

You can download my 'devioctl' tool: http://www.ltr-data.se/files/devioctl.zip
Type for example
CODE
devioctl geometry C:

to see what disk.sys returns for drive C: and then
CODE
devioctl geometry E:

to see what imdisk.sys returns for an ImDisk drive E:.

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
BTW, general expression: very good professional work (IMG:http://www.boot-land.net/forums/style_emoticons/default/thumbup.gif)


Thanks a lot! (IMG:http://www.boot-land.net/forums/style_emoticons/default/cheers.gif)

QUOTE (Alexei @ May 11 2007, 05:01 PM) *
Regarding server-side of the communication proxy: I'd be interested in complete specification (I'm thinking about implementing it on windows). Even better it would be to have an API to a user-written DLL that provides virtual I/O (server side).
I mean something like this:
virtual disk<-->proxy client<-->transport<-->proxy server<--(API)-->UserDLL<-->whatever


The structures of the headers of the I/O packets sent between client and server are defined in inc\imdproxy.h and there is a sample implementation of it for TCP/IP redirection in the devio sub-directory. Most of the interesting stuff goes in devio/devio.c. I keep a compiled exe at http://www.ltr-data.se/files/devio.exe too. The source for devio is pretty much POSIX-ish as it started off as a project for a FreeBSD Live CD but it should be fairly easy to start with that and extend it so that instead of opening a file or device it can call functions in a dll.
Go to the top of the page
 
+Quote Post
Alexei
post May 12 2007, 09:30 AM
Post #24


Advanced Member
***

Group: .script developer
Posts: 647
Joined: 30-August 06
Member No.: 283



@Olof,
Thanks for reg descriptions and everything (IMG:http://www.boot-land.net/forums/style_emoticons/default/thumbup.gif)

QUOTE (Olof Lagerkvist @ May 11 2007, 09:38 AM) *
QUOTE
Alexei @ May 11 2007, 05:01 PM
Also, you have a lot of useful comments - just mark them somehow, then automatically extract to the docs.

Yes, I have been thinking about generating docs from comments with Doxygen but it did not understand Windows driver code very well so... Do anyone know anything better?

I'd say, Doxigen is "too much". Your source is not a part of a big project. Writing your own tool to extract what you mark may be faster then looking for something suitable on the web (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif) You can use any markers inside comments, such as <<<......>>> and just
copy extracted text to a text file. Next, it can be a batch that assembles everything, adds html header/trailer, etc (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)

QUOTE (Olof Lagerkvist @ May 11 2007, 09:38 AM) *
QUOTE
Alexei @ May 11 2007, 05:01 PM
I propose a feature (for better compatibility):
For each image create not only \\.\PHYSICALDRIVEx (drive), but also \\.\PHYSICALDRIVEn (volume)...

I can see that this could be useful but it requires lots of modifications both to the driver logic itself and to the way it registers the new virtual disks on the system so this will probably not be implemented in a near future.

I'm afraid, lack of the drive may create problems during startup. We'll see. Anyway, it's not required to be the same driver. (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
It can be additional driver that provides the drive. Each new leyer of mods adds complexity that provokes bugs...

QUOTE (Olof Lagerkvist @ May 11 2007, 09:38 AM) *
Interesting ideas. I have thought about some kind of write-copy mode that stores changed parts of the image in a memory buffer or in some kind of "undo file". The reason that is a heavy task to implement is that it needs some kind of cluster-indexing thingy in the background, some place where the driver can look in a table and see wether to read requested data from the original image file or from another place. But it is definitely something that I will try to implement in the future.

I would use a bit mask for that (bit per cluster), probably in non-pageable.

QUOTE (Olof Lagerkvist @ May 11 2007, 09:38 AM) *
The current uninstall routines should leave no traces. Is there anything in particular you think of here?

I'll check it up and provide the list, if any (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)

QUOTE (Olof Lagerkvist @ May 11 2007, 09:38 AM) *
IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS should only be implemented by disk drivers for volumes on hard disk drives and there are no hard disk drives associated with the ImDisk "partitions".

That's what Startup may dislike. We'll see (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)

QUOTE (Olof Lagerkvist @ May 11 2007, 09:38 AM) *
The structures of the headers of the I/O packets sent between client and server are defined in inc\imdproxy.h and there is a sample implementation of it for TCP/IP redirection in the devio sub-directory. Most of the interesting stuff goes in devio/devio.c. I keep a compiled exe at http://www.ltr-data.se/files/devio.exe too. The source for devio is pretty much POSIX-ish as it started off as a project for a FreeBSD Live CD but it should be fairly easy to start with that and extend it so that instead of opening a file or device it can call functions in a dll.

Yes, protocol is pretty simple. I'll play with it (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
Meanwhile, could you please add COM port to the list of supported transports?
I have personal interest in this function, as I began virtualizing my environment. COM may be faster then TCP/IP for host/guest communications. I'd like to use IMDISK to let Guest to access Hosts's HDD partitions "directly".

(IMG:http://www.boot-land.net/forums/style_emoticons/default/cheers.gif)
Alexei
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post May 12 2007, 06:10 PM
Post #25


Advanced Member
***

Group: Developer
Posts: 122
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


QUOTE (Alexei @ May 12 2007, 11:30 AM) *
I'd say, Doxigen is "too much". Your source is not a part of a big project. Writing your own tool to extract what you mark may be faster then looking for something suitable on the web (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif) You can use any markers inside comments, such as <<<......>>> and just
copy extracted text to a text file. Next, it can be a batch that assembles everything, adds html header/trailer, etc (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)


Sound like a good way to go. Think I will do something like that.

QUOTE (Alexei @ May 12 2007, 11:30 AM) *
I'm afraid, lack of the drive may create problems during startup. We'll see. Anyway, it's not required to be the same driver. (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
It can be additional driver that provides the drive. Each new leyer of mods adds complexity that provokes bugs...


Yes, there is a risk that this will create problems during startup. But I think that, along with some other issues like incompatibility with the GUI defrag tool in Windows etc come from the problem that ImDisk does not implement interaction with the Volume Mount Manager, it acts more exactly like a disk driver did in Windows NT 4.0 and earlier.

To implement Volume Mount Manager support it would have to call the Mount Manager to notify it about new disks, create GUID values for each virtual volume, answer Mount Manager's callback ioctl codes (IOCTL_MOUNTDEV_x) etc and that makes the driver a lot more complex. Ken Kato's Virtual Disk Driver and Virtual Floppy Driver implement some support for these codes but not really exactly the way they should be implemented as the DDK documents say... I think I would like to do this in a correct manner if doing it at all. In most cases it is fully ok to not implement Volume Mount Manager support at all, or else implement everything that the Volume Mount Manager documents state as mandatory to support for a storage driver.

But as you say this could very well be implemented in a separate driver and that would keep the original ImDisk pretty much as it is right now for other purposes than where strict Volume Mount Manager support is really needed.

QUOTE (Alexei @ May 12 2007, 11:30 AM) *
I would use a bit mask for that (bit per cluster), probably in non-pageable.


Yes, something like that. The kernel has very easy-to-use support functions for large bit-fields because that is very common for filesystem drivers to use. But it still requires a lot modification and I think I will implement things suchs as image file offsets and delayed read-in from image file to RAM-disks first.

QUOTE (Alexei @ May 12 2007, 11:30 AM) *
Yes, protocol is pretty simple. I'll play with it (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
Meanwhile, could you please add COM port to the list of supported transports?
I have personal interest in this function, as I began virtualizing my environment. COM may be faster then TCP/IP for host/guest communications. I'd like to use IMDISK to let Guest to access Hosts's HDD partitions "directly".


COM ports are already supported... (IMG:http://www.boot-land.net/forums/style_emoticons/default/secret.gif) (At least, maybe...) (IMG:http://www.boot-land.net/forums/style_emoticons/default/wink.gif)

Actually this has hardly been tested at all but some kind of support for it is implemented. Example:

CODE
imdisk -a -t proxy -o comm -f "COM1: BAUD=256000 PARITY=N DATA=8 STOP=1" -m F:


This should connect to a server end of an ImDisk proxy through COM1. It initializes using BuildCommDCBAndTimeouts() so the syntax to the -f switch is like the mode com command. I have no sample server implementation for COM ports but the communication protocol is the same as over TCP/IP. Again, this is mostly not tested at all so anything could happen (almost) (IMG:http://www.boot-land.net/forums/style_emoticons/default/betasof.gif)
Go to the top of the page
 
+Quote Post
phox
post May 13 2007, 05:07 AM
Post #26


Advanced Member
***

Group: .script developer
Posts: 723
Joined: 8-August 06
Member No.: 134



I have a feeling that this post generates very interesting discussion about
potentially promising feature for further development of WinBuilder.

@Olof,

Could you please, publish some GUI for ImDisk, so that non command language
oriented members could taste goodies of your product.

Thank you.
Go to the top of the page
 
+Quote Post
jaclaz
post May 13 2007, 07:27 AM
Post #27


Finder
***

Group: Advanced user
Posts: 1,622
Joined: 14-July 06
Member No.: 2


Italy


phox,
FYI IMDISK already has a GUI, in the form of a .cpl file that is installed in Control Panel.

Nuno took a screenshot at it, you can see it here:
http://www.911cd.net/forums//index.php?sho...=19711&st=2

It covers all the normal operations.

jaclaz
Go to the top of the page
 
+Quote Post
TheHive
post May 13 2007, 08:15 AM
Post #28


Advanced Member
***

Group: .script developer
Posts: 2,134
Joined: 14-July 06
Member No.: 5



I tried in BartPE XPE and it works. Ofcourse using gui myself after paraglider mentioned it was in the control panel. Its very good.

I love that you can make it removable, CD type or HD type. Excellent!
I would be cool if some one created a virtual CDRW type.
Go to the top of the page
 
+Quote Post
phox
post May 13 2007, 01:39 PM
Post #29


Advanced Member
***

Group: .script developer
Posts: 723
Joined: 8-August 06
Member No.: 134



QUOTE (jaclaz @ May 13 2007, 07:27 AM) *
phox,
FYI IMDISK already has a GUI, in the form of a .cpl file that is installed in Control Panel.

It covers all the normal operations.

jaclaz


Thank you! I found it and it's great.
Go to the top of the page
 
+Quote Post
« Next Oldest · ImDisk · Next Newest »
 

3 Pages V  < 1 2 3
Fast ReplyReply to this topicStart new topic

Members Who Viewed Topic Today ()

 

Display Mode: Standard · Switch to: Linear+ · Switch to: Outline

Track this topic · Email this topic · Print this topic · Subscribe to this forum

RSS Lo-Fi Version Time is now: 3rd March 2008 - 08:17 AM

MKPortal ©2003-2006 mkportal.it