Mfs_Ftp: extract, archive, restore & transfer recordings

[rc3105]

the header chunk of each fsid tells you how many chunks of video it contains in bytes 12-15

from the mfstools 2 readme

-r scale
This option allows you to reduce the amount of RAM TiVo uses by
increasing the block size for the media storage created with the -x
option. The acceptable values are 0 to 4, corresponding to 1, 2, 4,
8 and 16 megabytes. The larger the value, the less RAM will be used
(and the faster menus will respond) but a (small) amount of storage
will be wasted by some recordings. At the extreme small end (-r 0)
some PVRs with a large amount of storage may not be able to perform
some tasks, such as self repair. (A.K.A. Green Screen) The default
is 2 (or 4 megabytes). Any TiVo created partitions are created with
a scale of 0 (or 1 megabyte). It is best to leave this option alone.


if you have a mfstools restore on a large drive or adjusted the mimimum mfs allocation size with the -r option then sometimes the fsid is actually larger than the size required for the chunks it contains. the space between the last chunk and the end of the fsid is the junk that falls through the cracks. it's just whatever happened to be there when the fsid was allocated. 98% of the time it only happens at the end of the last fsid.

since tcl is required for tivo db calls and runs on every flavor of os/hardware I tinker with I write tivo utils as tcl scripts. (it's also easy to link in platform specific c modules as functions for speed) if needed they can be "compiled" into an executable with a runtime lib like freewrap as I did with tmf2ty & tmf2mpg. porting is unnecesary for the mac since X includes tcl. the import filter algo is in the ty2fsid proc in mfs_ftp that starts around line 500 and in tmf2ty but disabled by default


--
Riley

[osetivo]

Originally posted by rc3105
RE: AW's teaser. if you extract encrypted recordings as tmf they CAN be restored to the same unit. (slightly easier than archiving mfstools backups that include recordings)

By "same unit", do you mean the same harddrive as well, or could you d/l and reload scrambled tmf on a fresh HD install? Can you do it with TY+?

[rc3105]

AW has a bee in his bonnet and noodged me 'n mbm to update our insert routines... (if only I were that young and energetic again!)

so far scrambled recs seem to be tied to the crypto chip. they're working on 3.1 and I'm focusing on 2.x. I'll be moving a cyrpto between boards this evening to pin down whether it's tied to the chip or the drive image. my $$$'s on the chip

scrambled ty+ should insert ok, but tmf is better for archiving scrambles as the "junk between the cracks" is preserved and that may be a factor in restoring scrambles


--
Riley

[AlphaWolf]

Originally posted by rc3105

so far scrambled recs seem to be tied to the crypto chip. they're working on 3.1 and I'm focusing on 2.x. I'll be moving a cyrpto between boards this evening to pin down whether it's tied to the chip or the drive image. my $$$'s on the chip

Are you still trying to get oprah to play?

BTW, when you say you are moving the chip, are you actualy talking about physicaly moving the chip? or cloning the serial number?

[rc3105]

simply cloning the sernum doesn't work so I'll be physically desoldering the !@#$@#$ surface mount crypto chip. gotta replace the tuners / volt reg & who knows what else on those boards anyway. should be fun in the root canal sense of the word

Originally posted by needo
How exactly does this work? I have a scrambled stream on my TiVo 4.0. and tried to download the .ty (Same error as before.) I then downloaded the .tmf without any problems, but it still choked on the tmf2ty conversion. And I ended up with an invalid .ty.

What am I missing?

scrambled recs can't be converted to mpeg on the pc (yet). they can be backed up, restored and played back on the EXACT same tivo, or they can be decrypted into regular ty in the tivo, then extracted & converted to mpg. mfs_stream can't read scrambles so to archive w/o decrypting you have to use mfs_tarstream to extract as tmf
speed issues - the biggie
here I thought it was something complex & it was just the !@#$% ideturbo module, figures. thanks mbm!

to check your tivo boot parms issue this command from bash

bootpage -q /dev/hda

you should get roughly this

IP address: 192.168.0.2
Primary boot partition: 3
Alternate boot partition: 6
Hostname: unnamed
Net boot kernel name: linux.px
Boot parameters: root=/dev/hda4 runideturbo=false
MAC address: bd:ca:43:47:75:90
RF channel: 0
Standby: 109
1

if the "Boot paramters:" line DOES NOT include "runideturbo=false" then you need to add it

note whatever your root parms are. "root=/dev/hda4" or "root=/dev/hda7" and update them like this

bootpage -P "your_current_root_parms runideturbo=false" /dev/hda

reboot, try an insert. should be MUCH faster

[tivowatcher]

Oh yeah - this is nice!

Here are some transfer examples once the runideturbo=false parameter is added:

TIVO -> TIVO: 416 KBs
PC -> TIVO: 409 KBs

Seems like if I let it queue up about 5-10 mins, I can watch the show "real time" as it streams from the source.

Congrats rc3105 you've got a winner!

P.S. This is using 2 SVR-2000's - which are Series 1 standalones.

[Will]

Originally posted by tivowatcher
PC -> TIVO: 409 KBs

Seems like if I let it queue up about 5-10 mins, I can watch the show "real time" as it streams from the source.

Even at that speed you really should be able to start watching after a minute or so (though my experience is all on DTivos).

[mini__me]

WOW, that is sooo much faster I am now having the problem of inserts running so fast they are affecting normal use

I have extract running around 1,500k while recording at best quality and inserts around 1,000k again while recording at best however this severely impedes on normal use I'll have to get experimenting on throttles.

LOVING IT

[osetivo]

Originally posted by mini__me
WOW, that is sooo much faster I am now having the problem of inserts running so fast they are affecting normal use

I have extract running around 1,500k while recording at best quality and inserts around 1,000k again while recording at best however this severely impedes on normal use I'll have to get experimenting on throttles.

LOVING IT

1000KB Wheew! glad to hear it. I saw that 400KB insert speed mentioned earlier, so I didn't want to get my hopes up.

Already did the bootpage cmd (from work), just can't wait to get home to test upload speeds.

As always thanks Riley (and of course mbm)

[cojonesdetoro]

I'm getting about 550-650KBS uploads and am more than satisfied with the Tivo's performace. Recordings and menus all look fine so far. That's a good enough speed for me. It seems like high quality recordings go in at 'real time' now. That is, 1 hour of video uploads in about an hour.

[osetivo]

Originally posted by osetivo
1000KB Wheew! glad to hear it. I saw that 400KB insert speed mentioned earlier, so I didn't want to get my hopes up.

Already did the bootpage cmd (from work), just can't wait to get home to test upload speeds.

As always thanks Riley (and of course mbm)

Much better, With both tuners on empty channels I'm now getting ~650KB/s up from 250. Geesh 1,000 KB/s, I'm still jealous.

[embeem]

The ideturbo module is intended to put the drives in "streaming media" mode; a few tests suggest that drive access actually slows down while ideturbo is loaded, adding runideturo=false to the boot params will prevent the ideturbo module from loading:

bash-2.02# bootpage -p /dev/hda
root=/dev/hda4
bash-2.02# bootpage -P "root=/dev/hda4 runideturbo=false" /dev/hda
Updated boot page on /dev/hda

Playing around with hdparm can lead to further performance gains.

Also, the version of mfs_import distributed with mfs_ftp is rather inefficient; a newer version is avaiable here: http://alt.org/forum/index.php?t=msg&th=55&start=20

[rc3105]

mfs_ftp is a basic ftp implmentation. some os's and proxy's can mount it as a remote filesystem, other's can't. try & few & see what works for you

setpri is there to prevent inserts from eating so many cpu cycles that menu's get sluggish. get it from alt.org, put it in /var/mfs_ftp/ & make it executable or don't worry about it. it's there to slow things down. the log just lets you know whether it was found/used

mbm's optomized mfs_import is a bit faster BUT it has to be renamed to mfs_stdinsert (or mfs_ftp edited appropriatly) and it only works with mfs_ftp versions > 1.2.8f it doesn't affect transfer speeds so much as lowering the cpu load for a given transfer rate (which lets you insert faster w/o impacting system performance)

if you add runideturbo=false to your boot parms you have to reboot the tivo before that takes effect. check to see if ideturbo is loaded with "cat /proc/modules" if it's not listed you're good

hdparms can make a big difference, this is safe for stock quantums "hdparm -c1 -d1 -m8 -S0 /dev/hda /dev/hdb" try it from bash before adding to rc.sysinit or rc.remote-login

increasing info(ithrottle) on line 21 slows inserts down. values can be 0 to 10

increasing info(insert_priority) accelerates inserts if setpri is available. values are 1 (lowest) to 99 (highest) 5 works nicely here

ithrottle & insert_priority will interact differently depending on your hd / network / running-hacks / sw-version / viewing-behavior / karma. the worst case scenario is a dtivo/tivonet/Q30 recording 2 shows & rewinding a third while inserting. even under those conditions I'm getting 500k/sec+ here. a standby dtivo/turbonet should run 1,200k/sec+ easily

don't try to watch scrambled recordings untill import is complete. depending on your noscramble module/settings it could cause a reboot

test your network / hdparms / insert speed & tweak your settings with the tivo in standby. once you find the peak rate then try a worst case scenario & re-adjust. dtivo style partition layout can make a big difference on slow drives like the stock quantums (see previous posts re: mfstools restore with the -p option)

[AVD]

Originally posted by rc3105

check your hdparms with hdparm -i /dev/hda and pay attention to MaxMultiSect. if you have dual drives you also need to do this for /dev/hdb--
Riley

I have a quantum and my maxmultisect=16, should I go with 8 or 16?

Do I have to reboot?

Thanks

Keep up the good work, thanks for working on this!

/dev/hda:
Model=QUANTUM FIREBALLlct15 30, FwRev=A01.0F00, SerialNo=314015418411
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=32256, SectSize=21298, ECCbytes=4
BuffType=3(DualPortCache), BuffSize=418kB, MaxMultSect=16, MultSect=off
DblWordIO=no, maxPIO=2(fast), DMA=yes, maxDMA=2(fast)
CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=58633344
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4


bash-2.02# hdparm -m8 /dev/hda

/dev/hda:
setting multcount to 8
multcount = 8 (on)
bash-2.02# hdparm -i /dev/hda

/dev/hda:
Model=QUANTUM FIREBALLlct15 30, FwRev=A01.0F00, SerialNo=314015418411
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=32256, SectSize=21298, ECCbytes=4
BuffType=3(DualPortCache), BuffSize=418kB, MaxMultSect=16, MultSect=8
DblWordIO=no, maxPIO=2(fast), DMA=yes, maxDMA=2(fast)
CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=58633344
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4

[rc3105]

AVD:

if the drive MaxMultiCount is 16 then use that. the point was not to set multicount to more than the reported MaxMultiCount for the drive. the hdparm settings are lost when you power-down or reboot the tivo. once you figure out what options you're going to use add them to a startup script somewhere