Digg this topic Add to my del.icio.us Submit to SlashDot 3 Pages V   1 2 3 >  
Reply to this topicStart new topic
> Need help creating custom virtual disk driver, Unused HDD space on multiple machines, presented as one drive
darkwhisperer
post Sep 6 2007, 06:49 AM
Post #1


Member
**

Group: Members
Posts: 11
Joined: 6-September 07
Member No.: 10,476


South Africa


Hi all,

Im writing an application thats going to use unused space on machines across our network, and present that space to the user as a "drive".

A local service on the machine will control the drive. Any attempts to read or write to the drive will be intercepted by or passed to the service. The service will query the database and get the information regarding where on the network this data is stored, and then proceed to either get or send the data to this location.

I was looking at the source code for ImDisk, to try to figure out how the disk space is presented as a drive to the system. But its a work in progress, and very difficult for me to figure out. (IMG:http://www.boot-land.net/forums/style_emoticons/default/frusty.gif)

Please can someone explain in more detail, or at least point me in the right direction regarding working on things at this level. Maybe a good site that explains how to write these drivers.

Thanks in advance for any help given. (IMG:http://www.boot-land.net/forums/style_emoticons/default/thumbsup.gif)
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post Sep 6 2007, 08:23 AM
Post #2


Advanced Member
***

Group: Developer
Posts: 120
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


Interesting project!

Maybe you could take a look at the proxy functions of ImDisk. Under the source tree there is a directory called devio with a sample proxy service that just listens to I/O requests over a TCP/IP socket and then uses an ordinary file as storage.

The actual I/O requests are sent by the ImDisk driver and they are really very easy, basically a code that indicates whether it is a read or write request, the size of the date and then the data itself. Because the requests are sent over an ordinary TCP/IP socket it should be possible for you to implement a service for it in practically any programming language without any particular knowledge of driver development.
Go to the top of the page
 
+Quote Post
darkwhisperer
post Sep 6 2007, 09:21 AM
Post #3


Member
**

Group: Members
Posts: 11
Joined: 6-September 07
Member No.: 10,476


South Africa


Hi,

Thanks for that! Looks like my problem may be solved... Just have to figure out how to listen over a TCP/IP socket (doesn't sound too tough), and then kidnap the read & write calls to your RAMDISK.

If Im using those same calls, what will happen to the RAMDISK? IE The actual space used in memory? Additionally, wont size be a problem? Say my available space is 4Tb. Will the driver have any problems with that amount of space? Or will it just send and receive the data, and not really care how things are stored on the backend?

Thanks for all your help! And for your totally great App!
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post Sep 6 2007, 09:30 AM
Post #4


Advanced Member
***

Group: Developer
Posts: 120
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


QUOTE (darkwhisperer @ Sep 6 2007, 11:21 AM) *
If Im using those same calls, what will happen to the RAMDISK? IE The actual space used in memory? Additionally, wont size be a problem? Say my available space is 4Tb. Will the driver have any problems with that amount of space? Or will it just send and receive the data, and not really care how things are stored on the backend?

Exactly, when you create an ImDisk device with redirection of the I/O requests no RAMDISK storage is allocated by the ImDisk driver. All it will do is taking the I/O requests from the filesystem for the virtual disk and redirect it to your application.
Go to the top of the page
 
+Quote Post
darkwhisperer
post Sep 6 2007, 01:22 PM
Post #5


Member
**

Group: Members
Posts: 11
Joined: 6-September 07
Member No.: 10,476


South Africa


Ok, cool.

Its seems that setting up TCPIP listening is a little tricky in C#, or it could be that Im just tired. (IMG:http://www.boot-land.net/forums/style_emoticons/default/logik.gif) I'll continue working on that tomorrow.

I have a question about the data thats set up when a drive is created...

If you set up a drive, via a proxy, is it still formatted?

Each person who logs in should have their own drive, with only their data on it. This will be handled (what belongs to whom) by the service. But can I create an image, as a formatted drive with the NTFS file structure, but no actual data, and set that up to use with the proxy, so that when the data is being accessed, the proxy deals with returning the data(which isnt on the image)? Does the idea make sense, or am I missing something? (IMG:http://www.boot-land.net/forums/style_emoticons/default/wacko.gif)
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post Sep 6 2007, 01:44 PM
Post #6


Advanced Member
***

Group: Developer
Posts: 120
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


QUOTE (darkwhisperer @ Sep 6 2007, 03:22 PM) *
Its seems that setting up TCPIP listening is a little tricky in C#, or it could be that Im just tired. (IMG:http://www.boot-land.net/forums/style_emoticons/default/logik.gif) I'll continue working on that tomorrow.

Try to search for some sample code... there are lots of examples on the web. It is a lot easier to start from something already written.
QUOTE (darkwhisperer @ Sep 6 2007, 03:22 PM) *
If you set up a drive, via a proxy, is it still formatted?

No, ImDisk never formats any kind of virtual disk or in any other way modifies the contents of it by its own. You can use the ordinary Windows ways to format the virtual disk.
QUOTE (darkwhisperer @ Sep 6 2007, 03:22 PM) *
Each person who logs in should have their own drive, with only their data on it. This will be handled (what belongs to whom) by the service. But can I create an image, as a formatted drive with the NTFS file structure, but no actual data, and set that up to use with the proxy, so that when the data is being accessed, the proxy deals with returning the data(which isnt on the image)? Does the idea make sense, or am I missing something? (IMG:http://www.boot-land.net/forums/style_emoticons/default/wacko.gif)

Should be possible. For example, when a new drive is first mounted you could fill it with an empty NTFS filesystem structue and present that to the driver.
Go to the top of the page
 
+Quote Post
darkwhisperer
post Sep 6 2007, 02:42 PM
Post #7


Member
**

Group: Members
Posts: 11
Joined: 6-September 07
Member No.: 10,476


South Africa


Ok, got the TCPIP socket listener working, but having a problem with setting up the drive. Command used:

imdisk -a -t proxy -o ip -f 127.0.0.1:4567 -m W:

Error:

The ImDisk Virtual Disk Driver was loaded into the kernel.
The ImDisk Virtual Disk Driver Helper Service was started.
Error creating virtual disk: The parameter is incorrect.

So Im guessing one of my parameters was wrong. Cant figure out which one it is tho. Is any response expected when the drive is to be mounted? When I run this, a smiley face and about 4 newlines appear on my Console as the data the listener received. Could it be that it needs to send some data back before the drive is initialized?
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post Sep 6 2007, 04:01 PM
Post #8


Advanced Member
***

Group: Developer
Posts: 120
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


QUOTE (darkwhisperer @ Sep 6 2007, 04:42 PM) *
So Im guessing one of my parameters was wrong. Cant figure out which one it is tho. Is any response expected when the drive is to be mounted? When I run this, a smiley face and about 4 newlines appear on my Console as the data the listener received. Could it be that it needs to send some data back before the drive is initialized?

Okay, the data sent between the ImDisk driver and your application should follow a special protocol. Basically, first the driver asks your application for some information (sends IMDPROXY_REQ_INFO) and your application should respond with an IMDPROXY_INFO_RESP structure filled with some information about the virtual drive, for example the size of it. After that, the driver sends either IMDPROXY_READ_REQ or IMDPROXY_WRITE_REQ structures and your application should carry out the request and respond with an IMDPROXY_READ_RESP or IMDPROXY_WRITE_RESP structure.

The definitions for the structures are in inc\imdproxy.h and an implementation sample in C++ goes in devio\devio.c.

Hope this helps! (IMG:http://www.boot-land.net/forums/style_emoticons/default/smile.gif)
Go to the top of the page
 
+Quote Post
darkwhisperer
post Sep 7 2007, 10:40 AM
Post #9


Member
**

Group: Members
Posts: 11
Joined: 6-September 07
Member No.: 10,476


South Africa


Hi,

Thanks for that, the deeper details are starting to make sense now, and they just bring more questions...

Basically if I create a byte array 16777216 large, and just write/read all requests to there, I'll have a 16meg RamDisk. As I format it, I'll get to see how the FileSystem is set up on the disk. That right?

The only problem Im having is that I cant find a definition for IMDPROXY_REQ_INFO in imdproxy.h. I suspect its probably just my lack of C++ understanding.

This is what I understand so far of imdproxy.h:

The driver can send me 5 requests:
IMDPROXY_REQ_NULL, IMDPROXY_REQ_INFO, IMDPROXY_REQ_READ, IMDPROXY_REQ_WRITE, IMDPROXY_REQ_CONNECT, which is set up in the enum.

Now below that I see READ, WRITE, CONNECT - request & response pairs for each. Also an INFO response(no request?).
Now all the requests send 3*ULONGLONG pieces of data, probably as one chunk. I read the first 8 bytes, to get the request code. Now how will I know which number means which request code?
Then say I know its a read request, to read from offset to (offset+length). My response is errorno (perhaps 0 if all is well, but what are the other errors?) + length + a string of bytes as the data itself. Is that right?
If it wants to write, will it be sending me request_code+offset+length+(string of bytes as data)?

I hope you forgive me these questions, but Im really not used to programming at this level.
Thanks for all your assistance.
Go to the top of the page
 
+Quote Post
Olof Lagerkvist
post Sep 7 2007, 04:26 PM
Post #10


Advanced Member
***

Group: Developer
Posts: 120
Joined: 27-April 07
From: Borås, Sweden
Member No.: 6,234


Sweden


QUOTE (darkwhisperer @ Sep 7 2007, 12:40 PM) *
Basically if I create a byte array 16777216 large, and just write/read all requests to there, I'll have a 16meg RamDisk. As I format it, I'll get to see how the FileSystem is set up on the disk. That right?

Correct.
QUOTE (darkwhisperer @ Sep 7 2007, 12:40 PM) *
The driver can send me 5 requests:
IMDPROXY_REQ_NULL, IMDPROXY_REQ_INFO, IMDPROXY_REQ_READ, IMDPROXY_REQ_WRITE, IMDPROXY_REQ_CONNECT, which is set up in the enum.

Now below that I see READ, WRITE, CONNECT - request & response pairs for each. Also an INFO response(no request?).

Agree, that is not very well explained. What it actually does is that it just sends IMDPROXY_REQ_INFO as an ULONGLONG. There is no more data needed than the request code itself in this case, therefore there is no request structure for it.
QUOTE (darkwhisperer @ Sep 7 2007, 12:40 PM) *
Now all the requests send 3*ULONGLONG pieces of data, probably as one chunk. I read the first 8 bytes, to get the request code. Now how will I know which number means which request code?

The request code will be one of the values defined in the enum you mention above. Compare the request code to them to find out which one it is.
QUOTE (darkwhisperer @ Sep 7 2007, 12:40 PM) *
Then say I know its a read request, to read from offset to (offset+length). My response is errorno (perhaps 0 if all is well, but what are the other errors?) + length + a string of bytes as the data itself. Is that right?

Correct. The error codes are in this case traditional Unix/Posix error codes (same as C/C++ libraries use). Practically, all you need is to return 0 if ok and a non-zero value otherwise.
QUOTE (darkwhisperer @ Sep 7 2007, 12:40 PM) *
If it wants to write, will it be sending me request_code+offset+length+(string of bytes as data)?

Correct.
QUOTE (darkwhisperer @ Sep 7 2007, 12:40 PM) *
I hope you forgive me these questions, but Im really not used to programming at this level.
Thanks for all your assistance.

No problem! Happy coding! (IMG:http://www.boot-land.net/forums/style_emoticons/default/cool.gif)
Go to the top of the page
 
+Quote Post
« Next Oldest · ImDisk · Next Newest »
 

3 Pages V   1 2 3 >
Fast ReplyReply to this topicStart new topic

Members Who Viewed Topic Today ()

 

Display Mode: Standard · Switch to: Linear+ · Switch to: Outline

Track this topic · Email this topic · Print this topic · Subscribe to this forum

RSS Lo-Fi Version Time is now: 27th February 2008 - 05:08 PM

MKPortal ©2003-2006 mkportal.it