PDA

View Full Version : Unified mfs_* tools (originally tridge vplay)



Jamie
11-21-2004, 11:56 PM
This is a "unified" version of vplay distribution that attempts to merge the assorted branches that have evolved over the years. This includes the mfs_* programs used by mfs_ftp, the tserver and vserver programs, and mfs_tmfstream, a tool used for extraction on a linux PC. The main goal was to share a common code base so that performance and bug fixes could benefit everyone.

These programs use considerably less CPU than the earlier builds I've found. That has two advantages: faster transfers and less interference with the TiVo software and UI. This (http://www.dealdatabase.com/forum/showthread.php?t=39328) thread will give you an idea of the current performance possible with this package and the 2.4.27 backport usb2 network drivers.

There are seven attachments:

mips - binaries for Series 2 machines
ppc - binaries for Series 1 machines
Linux-i686 - linux binaries for PC side extraction or for use with tserver
Darwin-Power-Macintosh - OS X binaries for extraction or for use with tserver
Cywin - Build for x86 windows PC running cygwin. This is for use as a vserver client. *NOT* for use with locally attached disks.
source
architecture independent files (README's and patches)


Some of the packages are pushed to the second post due to the 5 attachment post limitation.

Fetch the src package and read the README and CHANGES files.

The src package includes the limited documentation available (sorry I didn't improve the situation much there) and two patches to mfs_ftp. These patches should be considered experimental and have not been vetted by Riley. Please don't bother him if have mfs_ftp problems when using these patches; bother me instead. My patch improves insertion speed and eliminates the intermediate programs by using mfs_uberexport directly. JBuehl's patch uses expiration times exported from mfs_tmfstream when inserting. The patches can be applied on your TiVo with "patch < mfs_ftp.jamie.patch" while in your mfs_ftp directory, if you have the busybox patch program installed. If you want both patches, apply the jbuehl patch first, then the jamie patch.

Fetch the binary packages for the architectures you are interested in and untar them on your PC and/or TiVO(s). You'll get mfs_vplay_tserver/bin.<ARCH> subdirectories. You can either add this to your path, or copy the programs into your favorite bin directory already on your path.

To use with the unpatched mfs_ftp, copy mfs_{export,import,stream,tarstream,uberexport} to your mfs_ftp directory and make a symlink from mfs_import to mfs_stdinsert. You'll have to use "ln -sf" when you create the symlink to replace the original mfs_stdinsert, if you didn't already move or remove it.

To use with the patched mfs_ftp, apply the patches to mfs_ftp 1.2.9p and copy mfs_{uberexport,import} to your mfs_ftp directory.

For use with tytools, use the tserver program here.

Post support questions in the Newbie or Extraction, Insertion & Streaming Support forums.

License & Credits:


Released under the Gnu GPL v2, with the exception of the mfs_ftp.tcl patches, which are under Riley's mfs_ftp license.

Many people contributed to this package. First and foremost, Tridge, the original author. See the CHANGES files for a list of the contributers.

Modifications by:

adh, jamie, jbuehl, jdiner, jonbig, mbm, musclenerd, ppchacker, sn9, w_cb
and possibly others...

{Edit 2004/11/23
I recommend caution with insertions. When insertions are interrupted or aborted, a recording object stays in the Now Showing list that can be difficult to delete. See the support thread (http://www.dealdatabase.com/forum/showthread.php?t=39498) for details.
}

{Edit 2005/05/08
Darwin and CYGWIN bin packages pushed to the second post.}

Jamie
11-30-2004, 12:07 AM
Additional Attachments.

Jamie
12-13-2004, 08:25 PM
I posted a new build today. The packages are now bzip2 compressed to save a little more space.

The main thing that has changed is tserver: it now includes embedded code that generates the NowShowing list. This is faster, and also simplifies installation, since you only need the tserver program itself now and no longer need NowShowing.tcl.

tserver still supports the -s option to specify a NowShowing script, but if you use it you won't get the benefit of the new changes.

I've tested the new tserver with 4.0.1b and 5.1.1.b. jdiner tested with various 2.X and 3.X releases. Please let me know if you encounter any problems. There are two new programs that might be useful for debugging if you do encounter problems: NowShowing and mfs_tzoffset. NowShowing prints the NowShowing list to stdout (like mfs_streams, but in the TyTools format). mfs_tzoffset prints out the time zone offset as determined from mfs.

Jamie
03-15-2005, 10:52 AM
A new build has been posted in the first thread:

2005/03/14 Changes:

Daniel Gimpelevich (aka sn9): OS X Port
Jamie: C version of AlphaWolf's ciphercheck.tcl
DarkHelmet: Contributed program to aid in-place unscrambling on S1 hardware.
Jamie: Add an option (-f) to mfs_info to "fix" a broken superblock CRC.
Jamie: Use the schema from 7.1a-02.
Jamie: Fix mkpkg.sh so that file types match compression used.

Note that to stay under the 5 attachment limit I had to remove the noarch package. Everything that was in noarch is in the src package.

Jamie
03-16-2005, 02:45 PM
jimthev pointed out a memory leak in the NowShowing generation code used by tserver. This bug could cause a long running tserver to eventually run out of memory. The 20050316 build fixes this problem.

Jamie
03-27-2005, 03:18 AM
20050326

Jamie
Merge CHANGES into a single file.

Fix the broken -f (fix crc) option to mfs_info.

Add -s (scan) options to mfs_info that can scan for the highest
tyStream FSID and size needed for a backup to preserve
loopsets. This could be useful for determining
a reasonable -f or -l value for mfsbackup.

Fix a bug in mfs_uberexport that prevented small files from
being exported.

Jamie
04-19-2005, 09:28 PM
20050419

Fix tzoffset bug that caused problems with tserver on UK tivo's.

Fix get_tivo_version() so it doesn't exit if the version is unavailable for some reason. This allows mfs_uberexport to work even if /SwSystem/ACTIVE isn't present.

Jamie
05-08-2005, 04:46 AM
A new build has been posted 20050508. Highlights include inetd support, contributed by maxwellsdaemon, S1 cachecard 'burst mode' command generation support, contributed by JB6783, and a CYGWIN build.

This (http://www.dealdatabase.com/forum/showthread.php?p=221631#post221631) post from maxwells_daemon explains how to setup tserver & vserver to run under inetd. He describes an S1 setup, but I'm using it on a S2 using the inetd in AlphaWolf's All-In-One package. The only thing I might add is that you need to create an /etc/passwd file with a root account.

-----------------------------------------------------------------------------
20050507

Jamie

Added a mfs_burstcmds command line program that does the equivalent of
JB6783's php code. Extended the vserver protocol slightly to
allow mfs_burstcmds to work through a vserver connection.

Include CYGWIN compiled binaries. These are NOT intended to be
used with a directly attached disk. There are well know issues
with attaching a tivo drive to a windows 2K or XP machine: the
boot blocks of the disk get clobberd. Instead, use these
tools with a remote vserver:

On the tivo, run vserver, either directly or through inetd.

On a PC running a cygwin shell:

# Set the ip address of the tivo
export MFS_DEVLIST=":your-tivo-ip"

Then use mfs_uberexport, mfs_burstcmds, or other commands as though
the tivo drive was directly attached to your PC.

-----------------------------------------------------------------------------
20050501

JB6783
Add support for LIST_SECTORS tserver command: used with
contrib/burst/tserver.php to extract partition and sector info
for Series1 CacheCard burst mode transfers.

See contrib/README.burst for more information.

-----------------------------------------------------------------------------
20050429

maxwells_daemon
New tserver/vserver -l and -L options.
-l sends error messages to the syslog. -L sends all messages.
So we don't have to rewrite everything, log.h redefines fprintf etc
to call syslog() if use_syslog is set - otherwise it runs fprintf as if
log.h weren't there.

New tserver/vserver -i option. This allows these programs to be started
on demand from inetd. stderr and stdout are written to /dev/null, so
it's probably best to use -l (or -L) too. inetd is available as part of
GNU InetUtils. A PPC port is available from here
http://homepage.ntlworld.com/maxwells.daemon/tivo/ports.html

Fix load_devs which generated an unterminated device name, so could
randomly crash.

Fix error message in generate_1_NowShowing, which probably would have
crashed.

Add usage message for vserver -h. Update usage message for tserver -h.
-----------------------------------------------------------------------------

Jamie
06-17-2005, 10:28 PM
A new build has been posted in the first post in the thread. This is mainly a Mac OS X bug fix release. It should work correctly now with both disk images and locally attached disks. Although the Darwin package has a different date stamp, it's built from the same sources -- it just took me a while to get access to a Darwin machine for the final build.

Note that the project is now hosted on sourceforge (http://sourceforge.net/projects/mfs-utils). CVS there may be ahead of the latest release here.


20050604

Jamie
Fix device handling for OS_X: use DKIOGETBLOCKCOUNT32 to get the
device sizes. Handle bsd style disk device names (/dev/diskNsM...).

-----------------------------------------------------------------------------

20050515

Jamie
Allow a recursive mfs_dumpobj to continue after running into
invalid fsid references.

-----------------------------------------------------------------------------

20050512

Jamie
Move to sourceforge; rename to mfs-utils.

-----------------------------------------------------------------------------

Jamie
11-17-2007, 08:58 PM
Here's a quick a dirty port to mfs64 for the TiVoHD. This is only the mips binaries and only for the TiVoHD. On other platforms, keep using the old version.

I plan to release a merged version that supports both mfs32 and mfs64, but that seems to be a slow going project, and several people have asked for something sooner, so here it is.

chevyman601
07-22-2008, 08:00 PM
Here's a quick a dirty port to mfs64 for the TiVoHD. This is only the mips binaries and only for the TiVoHD. On other platforms, keep using the old version.

I plan to release a merged version that supports both mfs32 and mfs64, but that seems to be a slow going project, and several people have asked for something sooner, so here it is.

is this for the DirectivoHD (hr10-250) or just for the TivoHD SA

thanks!

ScanMan
07-22-2008, 08:12 PM
This is only the mips binaries and only for the TiVoHD. On other platforms, keep using the old version.I think that sums it up; hr10-250 is regular S2 - use the old version.

sylan
12-18-2008, 02:10 AM
Here's a quick a dirty port to mfs64 for the TiVoHD. This is only the mips binaries and only for the TiVoHD. On other platforms, keep using the old version.

I plan to release a merged version that supports both mfs32 and mfs64, but that seems to be a slow going project, and several people have asked for something sooner, so here it is.

Was a merged or x86 version ever released? The latest release on sourceforge says "September 2006".

Jamie
12-18-2008, 11:00 PM
Was a merged or x86 version ever released? The latest release on sourceforge says "September 2006".Nope. Sorry.


The mfs-64 source is in the sourceforge CVS with the tag "mfs-utils64". It should be straight forward to check it out and make an x86 (or x64) build for your distribution.

borrows123
05-17-2010, 07:18 AM
One of my friend is dealing this kind of things now a days...
Let me confirm from him then I will give you the authenticated answer....

Heinrich
06-11-2010, 06:37 PM
I thought that the s3 was very similar to the Tivo HD except for the "currently playing" LED on the front. But I had to install the Series 2 mips version of mfs-utils for TyTools to work. If I install the GEN05--netopt-xt3 kernel from http://www.dealdatabase.com/forum/showpost.php?p=299520&postcount=37 will that allow me to run the 64 bit one ? Is it faster?

Jamie
06-12-2010, 10:44 AM
No. A stock S3 has a 32bit MFS file system. The kernel has nothing to do with it. Only the TiVoHD and Premiere have 64bit MFS.

In theory you could build a 64bit MFS file system containing an S3 image using the mfs from scratch methods, but I can't think of any advantage to doing that.

Sbmocp
08-10-2010, 10:34 PM
I downloaded the 64-bit version, ftp'd it to a directory on my Tivo, and used the command from the directory:

tar -xjf mfs-utils.64.tar.bz2

and get the errors:

tar: bzip2: Cannot exec: No such file or directory
tar: Error is not recoverable:exiting now.

I can see the file with ls, so...what have I missed?

Jamie
08-11-2010, 01:00 AM
...

tar: bzip2: Cannot exec: No such file or directory
tar: Error is not recoverable:exiting now.

I can see the file with ls, so...what have I missed?tar -j requires bzip2. Do you have bzip2 on your tivo? If not, perhaps you can bunzip it on your PC before transferring it to your tivo.

Sbmocp
08-11-2010, 08:06 AM
tar -j requires bzip2. Do you have bzip2 on your tivo? If not, perhaps you can bunzip it on your PC before transferring it to your tivo.

That would explain it...no, I don't. Is it part of another tool package I should download and install first, or do I just need to get that program? Could you provide a link to the proper version for a TiVoHD?

Jamie
08-11-2010, 10:53 AM
That would explain it...no, I don't. Is it part of another tool package I should download and install first, or do I just need to get that program? Could you provide a link to the proper version for a TiVoHD?Just do the bunzip on your pc. Any linux distribution will have it, and it is in cygwin for windows.

Sbmocp
08-11-2010, 11:23 AM
Just do the bunzip on your pc. Any linux distribution will have it, and it is in cygwin for windows.

Okay--thank you. I suppose I'd just put in in any directory in the TiVo's path, preferably the hack file directory?

mike_s
08-11-2010, 01:24 PM
A x86 binary isn't going to run on your TiVo. Just bunzip2 on a pc, as Jamie said. If you don't have Linux or cygwin, you can get a windoze binary of bzip2 here (http://www.bzip.org/downloads.html). On a Mac, stuffit expander may work, but note that it will truncate file names greater than 31 chars - there is a darwinport of bzip2, though. Then binary FTP the files to your TiVo.

ScanMan
08-11-2010, 01:58 PM
FWIW, 7-zip (http://www.7-zip.org/) also supports packing/unpacking the bzip2 format on many platforms including Win32/64.

Sbmocp
08-11-2010, 04:05 PM
mike_s:

I thought about that and wondered how the version I downloaded would work on the TiVo if it came from a Windows distribution package. :confused:

Scanman:

I have 7-Zip on my Windows system and noticed that, when I unzip the file, it produces a .tar file. I thought I could ftp that up to the TiVo and was going to try doing so.

I can usually follow directions on how to do things with little trouble, but when it comes to Linux/Unix a lot of peripheral knowledge is required if procedures aren't spelled out. Thanks for the input and helping me to learn the finer points.

ScanMan
08-11-2010, 04:15 PM
I have 7-Zip on my Windows system and noticed that, when I unzip the file, it produces a .tar file. I thought I could ftp that up to the TiVo and was going to try doing so.Yes, that's probably the easiest way to do it!

AlphaWolf
11-29-2010, 01:22 AM
Where are the sources for the mfs 64 binaries posted? I don't see em on the sourceforge CVS

Jamie
11-29-2010, 01:49 PM
Source is in sourceforge CVS with tag "mfs-utils64": link (http://mfs-utils.cvs.sourceforge.net/viewvc/mfs-utils/mfs-utils/?pathrev=mfs-utils64).

AlphaWolf
11-29-2010, 06:03 PM
ah right thanks

nikole95.7
01-03-2011, 04:00 AM
I think that sums it up; hr10-250 is regular S2 - use the old version.

Was a merged or x86 version ever released? The latest release on sourceforge says "September 2006".

Jamie
01-03-2011, 02:34 PM
Was a merged or x86 version ever released? The latest release on sourceforge says "September 2006".Nope, but the 64 bit source is in sourceforge, so can check it out and compile it yourself.

mike_s
01-03-2011, 07:45 PM
Nope, but the 64 bit source is in sourceforge, so can check it out and compile it yourself.

You know that you're responding to a .sig spammer, don't you?

1fatboy
03-31-2011, 10:51 PM
No. A stock S3 has a 32bit MFS file system. The kernel has nothing to do with it. Only the TiVoHD and Premiere have 64bit MFS.


I've got a Tivo HD with a corrupt volume header that does the GSOD reboot loop. I was hoping to extract some shows off the failed drive using PC extraction. Do I need to build the mfs 64bit sources on Linux x86 to extract the streams?

I've read where Spike said a corrupt volume header error by mfs_info means the super block header is missing. I'm not sure if I'm wasting my time or should just cut my losses.

Jamie
04-01-2011, 12:15 AM
I've got a Tivo HD with a corrupt volume header that does the GSOD reboot loop. I was hoping to extract some shows off the failed drive using PC extraction. Do I need to build the mfs 64bit sources on Linux x86 to extract the streams?

I've read where Spike said a corrupt volume header error by mfs_info means the super block header is missing. I'm not sure if I'm wasting my time or should just cut my losses.Are your shows still encrypted? If so, I'm not sure PC side extraction is really that useful to you.

If they are unencrypted, then, yes, getting the source and compiling yourself is the way to go.

1fatboy
04-01-2011, 12:41 AM
Are your shows still encrypted? If so, I'm not sure PC side extraction is really that useful to you.

If they are unencrypted, then, yes, getting the source and compiling yourself is the way to go.

Yep, they're all unencrypted. I'll give it a try. Btw, I replaced the two disks with a single 2 Tb disk tonight. I'm running your custom kernel so thought it was a good opertunity to reduce chance of drive failure by 50% :D Its back up, now just to see how much can be recovered off the old disks. Thanks!

1fatboy
04-01-2011, 11:50 AM
I was able to get some shows, mostly the newly recorded ones. Unfortunitually, a lot of the older recordings have atleast 1 or 2 parts of the TMF that have a fsid of size 0. On the upside the mfs_utils 64bit build worked like a champ and I've got another way to extract. Again, thanks.

Jamie
04-01-2011, 12:56 PM
Glad you were able to recover something anyway.

I see that I have a local patch to export.c, that I think came from bcc [ref (http://www.dealdatabase.com/forum/showpost.php?p=295133&postcount=15)]. it relates to how the part size is determined. You could try it to see if it helps.

1fatboy
04-02-2011, 12:41 AM
Glad you were able to recover something anyway.

I see that I have a local patch to export.c, that I think came from bcc [ref (http://www.dealdatabase.com/forum/showpost.php?p=295133&postcount=15)]. it relates to how the part size is determined. You could try it to see if it helps.

I got pretty excited to see the patch, but unfortunately it had the same result (size=0). I might add some debug to the new method tonight to see what the Ty master header contains. I feel like the data "should" still be there, but really wont know for sure until we can see if the structs contain anything meaningful.

It was really cool to see some shows extract off, but the bad part was they weren't the shows I wanted most. That's how it usually works out though. :)

vandebo
10-23-2011, 06:37 PM
I wanted to learn the MFS format so I could investigate live streaming in progress recordings. I figured a good way to do that was to merge the 32bit and 64bit branches. It would be great if you could apply this to the sourceforge repro. Let me know if there are parts you'd like done a different way.

This patch is against the 64bit branch and merges all changes from head after the branch point. I tested the following commands using vserver on a TivoHD and a S1 Tivo: mfs_info mfs_ls mfs_dumpobj mfs_streams mfs_bitmap vplay

7342

--
Steve

Jamie
10-24-2011, 01:27 AM
Cool! Nice work! I'll have a look and see if there is anything I'd suggest you change. Otherwise, if you have a sourceforge account, I can set you up as a developer on the project and you can commit the changes yourself. It will be nice to have the two branches unified again. That's a cleanup project I never seemed to have time for.

Soapm
10-28-2011, 10:27 AM
I'm not getting the full Now Playing List when I launch TyTool9r18.zip on my PC. Would this be the thread to ask for help or should I use the tytool thread?

Jamie
10-28-2011, 03:51 PM
This is a thread in the FILES forum, so it is not the appropriate place to ask questions, it is a place to post files.

If it is TyTools related, you should probably start asking there, although as far as I know, no one is currently supporting that project.

It does not look like a support thread was ever created for mfs-utils, so either start a thread in the Newbie forum, or in the Extraction, Insertion & Streaming Support forum, depending on how technical the question is. A basic "I can't get this to work" question should probably be in the Newbie forum.

vandebo
10-30-2011, 11:30 PM
The two branches on sourceforge cvs have now been merged. Builds from 64 bit filesystems should be done from mainline. If you observer a confirmed regressions (works on the 64 bit branch or trunk before the merge, but not after the merge), report it here.

--
Steve

Jamie
10-30-2011, 11:39 PM
Great. Thanks!

I'll post updated binaries soon. I probably can't provide Mac binaries anymore, unless someone else wants to build them.

Soapm
02-05-2012, 07:38 PM
Nevermind, just found README.MFS_DEVICE

export MFS_DEVICE="/dev/sdc" is what I needed

spitfires
01-27-2013, 06:31 AM
Just in case anybody else ever tries it, the "64-bit merge" and subsequent versions won't compile for PPC (S1). The last version that I've managed to compile (untested) for PPC is mfs-utils-20050604

The latest cross-compiler I have for ppc is gcc version 2.8 and this barfs on the 64-bit merge code. I tried modifying the code to enable it to compile but the changes were just too many and I gave up.

Neither have I had any success with generating a x-compiler based on a newer version of gcc - I tried both gcc 3.4 and 4.1 but compiling gcc fails with seg faults.



(And before anyone says WTF, I would point out that S1 PPC TiVos are all that 50% of the UK population have available, so please don't tell me to get a newer one!)

vandebo
01-28-2013, 02:36 AM
Have you tried building your tool chain using crosstool? http://kegel.com/crosstool/

jkozee
01-30-2013, 12:50 AM
Here's a patch against the current sourceforge cvs main that should allow PPC compilation. With this patch, I was able to compile using the 2.8.1 pre-built x86 ppc toolchain from sourceforge tivotools. The two issues I found with the 64-bit merge code was that that pre C99 version of gcc did not like having the packed attribute declared between the struct/union tag and the name of the type (must be past the closing brace of a definition) and variables need to be declared at the start of a compound statement. Additionally, s1_unscramble.c required some changes due to function changes.