PDA

View Full Version : MFS_Insert >2GB failing, rebooting s2



bndt12
08-31-2005, 01:17 PM
I am posting a second time, sorry....

I am moving a ton of recordings from my old T60 to my new DSR708. I have the latest bins loaded on the S2 and am UL'ing from a Server 2003 PC to the S2. For some reason when I try to insert from this PC using mfs_ftp and SmartFTP client, a >2gb file gets to the 2gb mark and the S2 reboots! I have done may files <2gb with no issues AND have FXP'ed a number of unscrambled Dolby Digital recordings from the T60 to the S2 >2gb with no issues? All other mfs functions are working OK.

Any ideas? This is wearing me out! I have searched and researched but nothing...

Jamie
08-31-2005, 01:21 PM
I am posting a second time, sorry....

I am moving a ton of recordings from my old T60 to my new DSR708. I have the latest bins loaded on the S2 and am UL'ing from a Server 2003 PC to the S2. For some reason when I try to insert from this PC using mfs_ftp and SmartFTP client, a >2gb file gets to the 2gb mark and the S2 reboots! I have done may files <2gb with no issues AND have FXP'ed a number of unscrambled Dolby Digital recordings from the T60 to the S2 >2gb with no issues? All other mfs functions are working OK.

Any ideas? This is wearing me out! I have searched and researched but nothing...Here's some debugging hints: Look at the log files. A level 5 mfs_ftp port.3105.log file is always useful for mfs_ftp problems. Also /var/log/messages, /var/log/kernel, etc. Any error messages on the serial console before the reboot?

bndt12
08-31-2005, 01:42 PM
Here's some debugging hints: Look at the log files. A level 5 mfs_ftp port.3105.log file is always useful for mfs_ftp problems. Also /var/log/messages, /var/log/kernel, etc. Any error messages on the serial console before the reboot?

Thanks for the quick reply.

I will take a look at the logs but I do not have a serial console set up. To crank up MFS_FTP logging, is the setting in mfs_ftp.tcl?

Jamie
08-31-2005, 01:57 PM
Thanks for the quick reply.

I will take a look at the logs but I do not have a serial console set up. To crank up MFS_FTP logging, is the setting in mfs_ftp.tcl?look for info(dbl) in either mfs_ftp.tcl or settings.tcl

It might be worth the $5 for a serial console cable. It can be hard to debug things like this without it.

bndt12
08-31-2005, 09:18 PM
Thanks Jamie...

I'll run some additional tests when I get home this weekend.

bndt12
09-06-2005, 02:17 PM
I did an insert test back to my T60 and see the same 2GB issue but no reboot. Here is the section of the port.3105.log at level 5. This is following the 4th chunk transfer(2GB):

inserted 511 meg at 919k/sec
importing fsid 4295361 of size 0
bgerror invoked with error

" error writing "file47": broken pipe "

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p
info(tswv): 3.1.0c2-01-1-011
info(dbl): 0
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105


catch close lastsock val ""


In the FTP client log it thinks the transfer completed?

Any ideas?

bndt12
09-06-2005, 02:21 PM
BTW - These recordings are from an S1 and I have run them through ty1to2.exe to recode the non-Dolby audio streams to work on my new S2.

Jamie
09-06-2005, 02:35 PM
info(dbl): 0
This indicates that the debug level is 0, not 5. There should be a lot more info in a level 5 log. You might need to change it in both settings.tcl and mfs_ftp.tcl. I believe it is set in both places, and I can't remember which one "wins".

The "size 0" looks suspicious.

This is starting to sound familiar now. Have you tried a different ftp client on the PC side?

bndt12
09-06-2005, 02:45 PM
This indicates that the debug level is 0, not 5. There should be a lot more info in a level 5 log. You might need to change it in both settings.tcl and mfs_ftp.tcl. I believe it is set in both places, and I can't remember which one "wins".

The "size 0" looks suspicious.

This is starting to sound familiar now. I seem to recall an issue in mfs_ftp overflowing the size of a signed 32 bit int. I'll have to search around to see if I can find it again.

OK, I only set it in settings.tcl. I saw that "size 0" as well. I wonder if it has something to do with the ty1to2 app? I am investigating this as well.

bcc
09-06-2005, 02:50 PM
I suspect that ty1to2 has the same LFS problem (failure when seeking past 2GB boundary) that Jamie found in tymplex, since ty1to2 shares some of its code. This would result in the XML being written to the wrong point in such large files. Let me go recompile with _FILE_OFFSET_BITS set to 64...

Jamie
09-06-2005, 02:53 PM
I suspect that ty1to2 has the same LFS problem (failure when seeking past 2GB boundary) that Jamie found in tymplex, since ty1to2 shares some of its code. This would result in the XML being written to the wrong point in such large files. Let me go recompile with _FILE_OFFSET_BITS set to 64...Yeah, I just realized we were talking about ty1to2 generated streams and it might be that same issue again.... The size comes from the master chunk for the part, it it looks like it might not have been filled in.

bcc
09-06-2005, 07:04 PM
Actually I'm not sure how to fix this under microsoft visual c. I tried replacing lseek with _lseeki64(), but all seeks return -1 at that point...

bcc
09-06-2005, 07:55 PM
Never mind: my _lseeki64 test was failing simply because of a bad uint64_t define.

bcc
09-06-2005, 08:11 PM
Ok, fixed version over here: http://www.dealdatabase.com/forum/showpost.php?p=233975&postcount=65

bndt12
09-06-2005, 10:28 PM
Ok, fixed version over here: http://www.dealdatabase.com/forum/showpost.php?p=233975&postcount=65

This is awesome, thanks for the quick turnaround!

I am testing and will post my results.

BTW - Is the date off on your build machine? The new ty1to2.exe is dated 5/10/2005?

bcc
09-06-2005, 10:35 PM
BTW - Is the date off on your build machine? The new ty1to2.exe is dated 5/10/2005?Damn I somehow slipped dragging&dropping that file back to my linux box. I've now updated the package (and changed the vers. to 1.3) with the new .exe....

bndt12
09-06-2005, 11:59 PM
Damn I somehow slipped dragging&dropping that file back to my linux box. I've now updated the package (and changed the vers. to 1.3) with the new .exe....

OK, I tested with the posted 1.2 and it still failed so I will test with this one.

Thanks again!

bndt12
09-07-2005, 12:19 AM
Damn I somehow slipped dragging&dropping that file back to my linux box. I've now updated the package (and changed the vers. to 1.3) with the new .exe....

:rolleyes: Tested the 1.3 and am getting errors as it begins to process the ty+:

error: lseek failed

System error string: Invalid argument

I have a script i use to process so I am consistent on syntax and I used a source file that I used previously with the .exe from dated 5/10 so teh source file should be fine.

bcc
09-07-2005, 02:48 PM
I neglected to override the system's off_t definition to force it to be 64 bit. Fixed. Obviously I should have tested the results first. I tested it this time, and it works, tho I don't have a >2GB S1 recording handy at the moment to test completely.

bndt12
09-07-2005, 04:18 PM
I neglected to override the system's off_t definition to force it to be 64 bit. Fixed. Obviously I should have tested the results first. I tested it this time, and it works, tho I don't have a >2GB S1 recording handy at the moment to test completely.

Thanks! I won't be able to test it until tomorrow but I will post the results.

bndt12
09-08-2005, 04:15 PM
I neglected to override the system's off_t definition to force it to be 64 bit. Fixed. Obviously I should have tested the results first. I tested it this time, and it works, tho I don't have a >2GB S1 recording handy at the moment to test completely.

Crap, it failed again at 2GB. Here is the port.3105.log. Let me know if there is anything else I can supply to help.

Jamie
09-08-2005, 04:26 PM
Crap, it failed again at 2GB. Here is the port.3105.log. Let me know if there is anything else I can supply to help.This looks to me to be a different problem. In fact, I don't see any evidence in the port.3105.log file that the transfer failed at all. It appears to have successfully transfered 2560MB.

bcc
09-08-2005, 05:28 PM
Yes, looks like you successfully inserted "Atlantis The Lost Empire" at exactly 2.5GB in length. Since it's an exact multiple of tivo's max segment length, I suspect you did some manual unscrambling work, and maybe you have a problem with that?
PS: You could get a better digital copy from ripping the dvd... Just saying.

bndt12
09-08-2005, 06:54 PM
Yes, looks like you successfully inserted "Atlantis The Lost Empire" at exactly 2.5GB in length. Since it's an exact multiple of tivo's max segment length, I suspect you did some manual unscrambling work, and maybe you have a problem with that?
PS: You could get a better digital copy from ripping the dvd... Just saying.

At 10:11:47 it tries to do the last chunk insert and fails. There are only 4 actual transfers that are successful, not 5? Then later in the log it states the actual size of the transfer at 2147483648 k.

I use unscramble.o and when I insert or FXP without running ty1to2 they are fine(except no sound of course).

Doesn't it look like it is failing when trying to get the last stream/FSID?

bndt12
09-08-2005, 07:24 PM
I have some that are recorded unscrambled and I will test one of them next.

I'll post results...

bndt12
09-08-2005, 11:07 PM
OK, I did a recording that was never scrambled and it worked great.

I just can't figure out why when I run a scrambled recording through unscramble.o, all indications are it is clean. I will play after exporting to PC and will play on the S2 but with no sound.

Any other ideas?

Thanks again for all the help!

bcc
09-09-2005, 01:55 AM
I just can't figure out why when I run a scrambled recording through unscramble.o, all indications are it is clean. I will play after exporting to PC and will play on the S2 but with no sound.In the unscramble.o instructions, it says "Wait at least 10 seconds between extractions". But I suspect this should really say wait 10 seconds between extraction of each segment (FSID). When I last used unscramble.o, it only unscrambled some of the FSID segments in particular recordings (usually just the first one), and I believe it's because of this. Gotta go back and re-extract a couple recordings because of this myself actually.
You can always rmmod unscramble.o and test the unscrambled recording for complete playback-ability before bothering to transfer the recording off.

bcc
09-09-2005, 02:01 AM
Also, if your unscrambling was botched at an FSID boundary, ty1to2 should have reported some errors, and the converted .ty would have been shorter than the original .ty. (because ty1to2 would have skipped the chunks it couldn't read).

bndt12
09-09-2005, 01:41 PM
I no longer have the S1 connected to a TV so I cannot test playback. I have however been through an entire show after unscrambling, on PC and on <2GB recordings on the S2. Unscramble.o does not allow FSID unscrambling, just on the show itself.

I do see ty1to2 rebuilding chunks when it runs but I believe this is mentioned in the readme as normal? The resilting ty+ is usually a bit larger than the original.

bcc
09-09-2005, 06:54 PM
Unscramble.o does not allow FSID unscrambling, just on the show itself.Each segment within a show has separate scrambling keys. I don't know exactly what you typed to unscramble, but unscrambling is performed 1 FSID at a time.
I do see ty1to2 rebuilding chunks when it runs but I believe this is mentioned in the readme as normal? The resilting ty+ is usually a bit larger than the original.That is normal. My comment about short .ty's was in regards to your failure situation. If you checked the file size, before insertion, and found it to be wrong, you'd be able to narrow down your problem quite a bit.

bndt12
09-10-2005, 10:01 AM
I did some further testing and found some new info:

> The insert fails on teh last full size, 512MB, chunk. If teh recording was 2.7GB then it would stop at 2.5GB

> Using unscramble.sh, which uses mfs_stream and mfs_insert, it continues to fail following ty1to2.exe but not before. If you insert prior to running ty1to2 it works.

> Using unscramble.o, loading module and playing a few seconds of recording, it fails following ty1to2.exe but not before. If you insert prior to running ty1to2 it works.

> Using s1_unscramble(mfs_unscramble) seems to work fine after running through ty1to2. I have inserted back to the s1 successfully and am testing insert to s2 now. This is available in the contrib folder in the mfs_utils package. I wrote a script to allow multiple FSID's to be processed:

#!/bin/sh
for fsid in $*
do
./s1_unscramble $fsid
sleep 3
done

I'll post the final results once I get a successful s2 insert.

Thanks for everyone's help!

bcc
09-10-2005, 02:58 PM
Can't tell what's really going on from your description. Would need more details including ty1to2 program output, file sizes before&after. Perhaps your older unscrambling tools are creating the recordings with some surprising holes. Might have to troubleshoot what you've really got going on there, and that'd probably require copies of your files. Sounds like you have a working method and so the troubleshooting wouldn't be worth it.

bcc
09-11-2005, 02:27 AM
BTW, I finally tried this myself on a recording >2GB. Worked no problem (well actually I had endless problems with descrambling which may be relevant). Correct results look like:
% ls -l oscar-wilde*
-rw-r--r-- 1 root root 2231373721 Sep 10 20:53 oscar-wilde.ty
% ty1to2 -i oscar-wilde.ty -o oscar-wilde-out.ty
Warning: Skipping chunk 17014 due to timestamp discontinuity
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:8191
Generating master chunk for chunks 8192:12287
Generating master chunk for chunks 12288:16383
Generating master chunk for chunks 16384:17789
Recording is 8449 seconds, 17790 chunks
% ls -l oscar-wilde*
-rw-r--r-- 1 root root 2331774842 Sep 10 20:58 oscar-wilde-out.ty
-rw-r--r-- 1 root root 2231373721 Sep 10 20:53 oscar-wilde.ty
% I suspect in your case you saw more than 1 warning (probably due to descrambling issues), and the master chunks are not separated with intervals of 4096. In the above, you can see the numbers increasing by 4096 at each line. If the master chunks are not spaced this way, you probably had more serious errors in the recording, and ty1to2 may forget to build the last master chunk in such cases. Which would lead to insertion problems.