How to increase sorting speed and reading of *.mcd files?

View previous topic View next topic Go down

How to increase sorting speed and reading of *.mcd files?

Post  rrenteri on Thu Mar 18, 2010 6:36 am

Dear MCS or any other users with advice,

We would like any advice to speed up our sorting process of spikes recorded from mouse retinas. We use Offline Sorter v3 from Plexon to sort MCD data files. The process of opening channels is quite slow because the data files are relatively large (multi-GB).

With a SATA2 hard drive, we reached a top speed of 40 MB/s. We installed a solid state drive, Intel X25-M G2, but this only increased that rate to 60 MB/s. We were hoping to get closer to the >200 MB/s theoretical speed.

The software reads each channel from disk using the MC_Stream.dll. This process seems to happen in 1KB blocks, but we're not sure.

Is there a way to increase the block size or some other way to increase the speed that Offline Sorter can get the data into memory? Iím asking here because this seems to be a limitation due to MC_Stream.dll and not due to Offline Sorter. If that is incorrect, please let me know. The drives are formatted under WinXP with a 4KB allocation block size.

(Alignment of waveforms is another bottleneck, and we're increasing CPU power for that.)

Thanks!
Reneí Renteria
University of Texas Health Science Center at San Antonio


Last edited by rrenteri on Thu Mar 18, 2010 6:38 am; edited 1 time in total (Reason for editing : (formatting))

rrenteri

Posts : 11
Join date : 2008-07-15

View user profile

Back to top Go down

Re: How to increase sorting speed and reading of *.mcd files?

Post  Hans MCS on Mon Mar 22, 2010 1:50 am

rrenteri wrote:
Is there a way to increase the block size or some other way to increase the speed that Offline Sorter can get the data into memory? Iím asking here because this seems to be a limitation due to MC_Stream.dll and not due to Offline Sorter. If that is incorrect, please let me know. The drives are formatted under WinXP with a 4KB allocation block size.
Thanks!

There is no internal block size to adjust in MC_Stream.dll. All reading is done with the standard library functions.
I have loaded a 1 GB file with spikes into offline sorter in just less than 50 s. I can't comment on offline sorter, but I suppose that the data are in memory after loading, so that opening a channel should not be a problem of disk speed anymore

regards
Hans

Hans MCS

Posts : 16
Join date : 2008-08-19

View user profile

Back to top Go down

Re: How to increase sorting speed and reading of *.mcd files?

Post  rrenteri on Wed Mar 24, 2010 6:31 am

Thank you. We're still trying to find the bottleneck. It seems that the sorter reads the file from disk but only to get information on number of waveforms and channels. Then it reads from disk again when asked to open an individual channel. It seems that processing happens after each block of disk reading as the sorter uses MC_Stream.dll to decode the mcd file. Or something like that.

Our goal is to speed up all the aspects of sorting. For example, aligning waveforms is a major bottleneck, and that is mostly CPU-limited from what we can tell. So we're moving to an overclocked i7-920 PC.

Best,
Rene'

rrenteri

Posts : 11
Join date : 2008-07-15

View user profile

Back to top Go down

Re: How to increase sorting speed and reading of *.mcd files?

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum