Tivo drive partitioning

[xjcrazed]

Is there a thread that discusses best practices for drive partitioning on a hacked Tivo? I've tried several searches, but can't seem to narrow the results enough to find the type of info I'm looking for.

Rob

[Narf54321]

Use Google, but also add the term site:dealdatabase.com which can yield better results than the internal search.

Anyway, you're probably looking for search terms pdisk and tivopart.

[xjcrazed]

First of all, thank you for the suggestion. Using pdisk and tivopart to search definitely returned better results. But I'm still not quite finding what I'm looking for. Let me give you more specific examples:

Here's an excerpt from Tiger talking about MFS Tools (http://www.tivocommunity.com/tivo-vb/showthread.php?p=2651877&&#post2651877):
Starting with the Series 1 DirecTiVo, all new models come with two MFS pairs on the first drive, of roughly equal size. The drive is then layed out with the MFS media (The shows) at either end of the drive, with the OS and application data in the middle. The application data is accessed very regularly, and my theory is that putting it in the middle is supposed to cut down on seek times by rarely having a seek cross more than half the drive.

I've learned that the PROM on some tivos can't see past 137GB and therefore it would be a bad idea on a large drive to have the kernel partitions >137GB on the drive. Does anyone know which models this is a problem for?

Also, if you are monte'ing from the 3.1.1c kernel, which is the recommended way to monte unless I'm missing something, the partition that your custom kernel is on would also have to be <137GB. Or you would have to monte from the 3.1.5 or 4.0.1a kernels, which last I read isn't stable. Has anything changed in respect to this?

Basically I'm trying to come up with an optimal partition layout before I start hacking. Does anyone know of any other threads that debate the merits of different partition layouts and their effects?

Rob

[fantmn]

I have been doing mine like the layout found in this thread (http://www.dealdatabase.com/forum/showthread.php?t=47356).
I need to update with my final technique on the post. Based on the some of the discussion in that thread I setup a 2GB partition (lots of extra space) as a work area to hold hacks, etc. It is on partition 5 and is in ext3 format. The only downside is that a mfsbackup DOES NOT save the workspace so you have to do additional work to save.
It seems that the stock layout work well for most folks. It depends on how strong your Linux skills etc. are and ow much reading you will do. There is a thread somewhere around here that discusses the best place to put hacks so they do not disappear, but I have not been able to find it.

[Narf54321]

I'm not really sure about what you are looking for. I'm also not familiar with PROM limitations, but yes you want the boot (kernel) partitions early on the drive.

The partition map tivo expects are in particular order, beginning with an Apple-style map. The MBR holds bootpage settings. /dev/hda8 is swap space (expanded to 127MB) and /dev/hda9 is the /var partition.
MFS usually starts on partition 10 in pairs, and has a small "App" followed by a larger "Data" partition.
Here is a partition listing from my 80hr tivo expanded to a 120GB drive (partitions 14&15 are the new expansions).

Backroom Tivo> pdisk -l

Partition map (with 512 byte blocks) on '/dev/hda'
#: type name length base ( size )
1: Apple_partition_map Apple 63 @ 1
2: Image Bootstrap 1 4096 @ 70056000 ( 2.0M)
3: Image Kernel 1 4096 @ 70060096 ( 2.0M)
4: Ext2 Root 1 262144 @ 70064192 (128.0M)
5: Image Bootstrap 2 1 @ 70326336
6: Image Kernel 2 8192 @ 70326337 ( 4.0M)
7: Ext2 Root 2 262144 @ 70334529 (128.0M)
8: Swap Linux swap 260096 @ 70596673 (127.0M)
9: Ext2 /var 262144 @ 70856769 (128.0M)
10: MFS MFS application region 524288 @ 71118913 (256.0M)
11: MFS MFS media region 70055936 @ 64 ( 33.4G)
12: MFS Second MFS application region 524288 @ 71643201 (256.0M)
13: MFS Second MFS media region 88046592 @ 72167489 ( 42.0G)
14: MFS New MFS Application 1024 @ 160214081
15: MFS New MFS Media 74219520 @ 160215105 ( 35.4G)
16: Apple_Free Extra 7023 @ 234434625 ( 3.4M)

[PlainBill]

First of all, thank you for the suggestion. Using pdisk and tivopart to search definitely returned better results. But I'm still not quite finding what I'm looking for. Let me give you more specific examples:

Here's an excerpt from Tiger talking about MFS Tools (http://www.tivocommunity.com/tivo-vb/showthread.php?p=2651877&&#post2651877):


I've learned that the PROM on some tivos can't see past 137GB and therefore it would be a bad idea on a large drive to have the kernel partitions >137GB on the drive. Does anyone know which models this is a problem for?

Also, if you are monte'ing from the 3.1.1c kernel, which is the recommended way to monte unless I'm missing something, the partition that your custom kernel is on would also have to be <137GB. Or you would have to monte from the 3.1.5 or 4.0.1a kernels, which last I read isn't stable. Has anything changed in respect to this?

Basically I'm trying to come up with an optimal partition layout before I start hacking. Does anyone know of any other threads that debate the merits of different partition layouts and their effects?

Rob

There might have been a limitation on Series 1 systems, all Series 2 proms are fully capable of handling drives > 137 Gig. There IS a potential problem when expanding a small (40 Gig) image to a large (> 250 Gig) drive using MFStools when monteing. MFStools might decide to put the new mfs partition at the beginning of the drive, moving the software partitions beyond the 137 Gig boundary. This can readily be avoided by restoring the image in one step, then adding the extra space as a second step.

PlainBill

[Jamie]

all Series 2 proms are fully capable of handling drives > 137 Gig.Are you sure about this? My (unverified) belief is that older series 2 models can't boot a kernel partition installed above the 137GB mark. Newer S2 models (HDTivo,264,275) and all S2.5 models can. I'd have to do some testing to verify this for sure.

Note that once you've booted an LBA48 aware kernel, the fact that the PROM itself can't address above 137GB is irrelevant -- it's the running kernel that matters.

[xjcrazed]

It seems that the stock layout work well for most folks. It depends on how strong your Linux skills etc. are and ow much reading you will do. There is a thread somewhere around here that discusses the best place to put hacks so they do not disappear, but I have not been able to find it.

That's just it, if I'm using a 300GB drive and want to precisely mimic the stock layout then the first MFS media partition will push the kernel partitions and root fs partitions to about the 150Gb mark. That has the potential to create two booting problems.

First, if the prom can't see that far onto the drive, it won't be able to find a bootable kernel. Second, if I want to monte and have to use the 3.1.1c kernel to monte from, it will fail because it won't be able to find the custom kernel.

I consider myself to have competent linux skills (new to the whole hacking thing though), and have already done lots of reading, searching, etc. This is a great forum, there are just some inherent problems to locating relevent information in this particular medium.

I saw that link once upon a time. I wasn't ready to read that information when I found it and it looks like I forgot to bookmark it. Thus demonstrating my statement about the medium. Thanks for the other link.

Rob

[xjcrazed]

I'm not really sure about what you are looking for.

I'm looking for a discussion about the pros and cons of different partition layouts as they pertain to Tivos. For instance, on a workstation the most commonly accessed information should ideally be located at the beginning of a drive. That's what defrag programs do and how they help system performance. Initially I was asking for help locating a thread detailing how partitioning affects Tivos, but its starting to look like I've started one. :D

The partition map tivo expects are in particular order, beginning with an Apple-style map.

The layout you provided is in numerical order, but to further illustrate my point, here is your same partition layout as it physically appears on the disk:

#: type name length base ( size )
1: Apple_partition_map Apple 63 @ 1
11: MFS MFS media region 70055936 @ 64 ( 33.4G)
2: Image Bootstrap 1 4096 @ 70056000 ( 2.0M)
3: Image Kernel 1 4096 @ 70060096 ( 2.0M)
4: Ext2 Root 1 262144 @ 70064192 (128.0M)
5: Image Bootstrap 2 1 @ 70326336
6: Image Kernel 2 8192 @ 70326337 ( 4.0M)
7: Ext2 Root 2 262144 @ 70334529 (128.0M)
8: Swap Linux swap 260096 @ 70596673 (127.0 9: Ext2 /var 262144 @ 70856769 (128.0M)
10: MFS MFS application region 524288 @ 71118913 (256.0M)
12: MFS Second MFS application region 524288 @ 71643201 (256.0M)
13: MFS Second MFS media region 88046592 @ 72167489 ( 42.0G)
14: MFS New MFS Application 1024 @ 160214081
15: MFS New MFS Media 74219520 @ 160215105 ( 35.4G)
16: Apple_Free Extra 7023 @ 234434625 ( 3.4M)

i.e. partition #11 physically resides on the disk before partitions 2-10. In my previous post I quoted Tiger's theory for why they did this, and I'm hoping to generate further comment.

Rob

[xjcrazed]

There might have been a limitation on Series 1 systems, all Series 2 proms are fully capable of handling drives > 137 Gig.

Handling the drive yes. But can it boot a kernel if it physically resides past the 137GB mark?

This was a common problem with earlier linux systems when multi-booting when linux resided after the other OSes on the disk. Linux didn't have any problems recognizing and using large (for the time) drives, but there was an 8GB(13GB? I can't remember anymore) limitation that made it so the bootloader (grub, lilo) had to reside physically on the disk below that limit. This isn't a problem anymore, (unless you're on older hardware) but you'll still get warnings on some distribution installs that the system might not boot.

Rob

EDIT: I believe that based on this post (http://www.dealdatabase.com/forum/showpost.php?p=180437&postcount=6) Jamie was correct in his recollection that only newer S2 units support booting a kernel past 137GB.

[xjcrazed]

Are you sure about this? My (unverified) belief is that older series 2 models can't boot a kernel partition installed above the 137GB mark. ... I'd have to do some testing to verify this for sure.

If a Hughes SD-DVR40, or RCA DVR40 count as older S2s then I'd be able to test.

Note that once you've booted an LBA48 aware kernel, the fact that the PROM itself can't address above 137GB is irrelevant -- it's the running kernel that matters.

Speaking of, has there been any progress on monte'ing from the 3.1.5 kernel?

Rob

[Jamie]

Speaking of, has there been any progress on monte'ing from the 3.1.5 kernel?It works for many. I haven't looked at it in a while, but in my past tests, it sometimes worked and sometimes didn't depending on the destination kernel.

FWIW, on the original question: mfstools 2.0 "optimizes" the partition layout before expanding, so if you start with, say, a minimal mfs image, and expand it onto a much larger disk, your partition layout won't be "optimal" in the sense Tiger described.

In practice, I haven't found partition layout to make much difference, but I haven't done careful benchmarks comparing performance with different layouts either.

I tend to prefer to have partitions laid out on disk in order by partition number, since it makes it easier to coallesce (http://alt.org/forum/index.php?t=msg&th=124&start=0&rid=0&S=4c013a9bbca80712dd90adfd491a3b7d#msg_num_2) partitions if you want to expand later and have already reached 16 partitions.

[xjcrazed]

FWIW, on the original question: mfstools 2.0 "optimizes" the partition layout before expanding, so if you start with, say, a minimal mfs image, and expand it onto a much larger disk, your partition layout won't be "optimal" in the sense Tiger described.

True. He wanted to add that functionality to MFS Tools 3.0. But it shouldn't be too difficult to set up the partitioning by hand to get an "optimal" setup. The question then becomes, what is "optimal"? Is the way Tivo sets up the drives "optimal"? And how should that be modified, if at all, for a hacked system?

In practice, I haven't found partition layout to make much difference, but I haven't done careful benchmarks comparing performance with different layouts either.

Given your knowledge of Tivos could you speculate on what might be best? For instance, on a S2 DTivo, lets say we're recording two shows simultaneously and watching a third previously recorded show. Or for that matter just watching a previously recorded show because the Tivo will still be writing the buffers.

That's a lot of disk access going on. Add an mfs_ftp or MRV transfer on a hacked system and things get even crazier. Ideally we'd want all those files close together on the disk so the head doesn't have to move far. But how does MFS deal with spreading files among multiple partitions? And within the partition how does it deal with fragmentation and reclaiming space?

Maybe it doesn't matter, but often knowing the high level use enables a better low level setup. In case you couldn't tell this is all coming from a sysadmin point of view.

I tend to prefer to have partitions laid out on disk in order by partition number, since it makes it easier to coallesce (http://alt.org/forum/index.php?t=msg&th=124&start=0&rid=0&S=4c013a9bbca80712dd90adfd491a3b7d#msg_num_2) partitions if you want to expand later and have already reached 16 partitions.

Speaking of, What tool would one use in order to edit the superblock? I'm guessing from the context one would just point a hex editor at the partition? Thanks for your input.

Rob

[Jamie]

...
Given your knowledge of Tivos could you speculate on what might be best? For instance, on a S2 DTivo, lets say we're recording two shows simultaneously and watching a third previously recorded show. Or for that matter just watching a previously recorded show because the Tivo will still be writing the buffers.

That's a lot of disk access going on. Add an mfs_ftp or MRV transfer on a hacked system and things get even crazier. Ideally we'd want all those files close together on the disk so the head doesn't have to move far. But how does MFS deal with spreading files among multiple partitions? And within the partition how does it deal with fragmentation and reclaiming space?

Maybe it doesn't matter, but often knowing the high level use enables a better low level setup. In case you couldn't tell this is all coming from a sysadmin point of view.You really have no control how tivo allocates space in the media regions. There should be ample disk bandwidth and seek capacity to support writing two SD streams (one captured from the tuner; one coming in from the network) and reading two streams (one to the tv; one to the network) all at once. Given the the I/O transfers are large (at least one 128KB chunk at a time), even if you had to do a full end-to-end seek for each transfer, there is ample disk performance capacity. HD might be a little more of a challenge, but mainly for transfer bandwidth, not seeks. Larger buffer sizes and minimum continguous allocation units can always reduce the need to seek as often. So my answer remains "it doesn't matter", at least to me.Speaking of, What tool would one use in order to edit the superblock? I'm guessing from the context one would just point a hex editor at the partition? Thanks for your input.I do it with dd, but it's easy to screw up so a hex editor may be safer.

[PlainBill]

Suggestion for the OP: Look at the partition map NARF54321 posted. Note that all non-mfs partitions take up less than 1 Gig of the drive. Remember also that the original MFS partitions will NOT be resized. If you expand a 40 Gig image to a 300 Gig drive you are going to wind up with two small mfs partitions totaling about 40 gig and one of 260 Gig. You can't predict where the TiVo is going to put recordings, so don't sweat it. Worry about important things, like what your wife is going to say if you lose one of her shows.

PlainBill

[xjcrazed]

You really have no control how tivo allocates space in the media regions.

Yes, but if we know any basics about how the tivo uses the space it might be possible to partition the drive to facilitate MFS transactions. And if we can optimize disk access it should also increase overall system speed.

There should be ample disk bandwidth and seek capacity to support writing two SD streams (one captured from the tuner; one coming in from the network) and reading two streams (one to the tv; one to the network) all at once. Given the the I/O transfers are large (at least one 128KB chunk at a time), even if you had to do a full end-to-end seek for each transfer, there is ample disk performance capacity. HD might be a little more of a challenge, but mainly for transfer bandwidth, not seeks. Larger buffer sizes and minimum continguous allocation units can always reduce the need to seek as often. So my answer remains "it doesn't matter", at least to me. ...

You're probably right, even with things completely optimized it might not make much difference. I just wanted to discuss it with those who have more knowledge and experience than myself.

Rob

[xjcrazed]

Suggestion for the OP: Look at the partition map NARF54321 posted. Note that all non-mfs partitions take up less than 1 Gig of the drive. Remember also that the original MFS partitions will NOT be resized. If you expand a 40 Gig image to a 300 Gig drive you are going to wind up with two small mfs partitions totaling about 40 gig and one of 260 Gig. You can't predict where the TiVo is going to put recordings, so don't sweat it.

True, if you rely on MFS Tools to do the partitioning for you. But if you manually partition and/or coalesce the MFS partitions I haven't seen anything that would keep you from say having just one pair of MFS partitions on a large drive. Or, trying to more precisely match what originally existed on the Tivo drive. The possibilities are endless, I'm just trying to figure out if there are any benefits to one setup over another.

Worry about important things, like what your wife is going to say if you lose one of her shows.

She'd kill me. :eek: But, being in the IT profession, I wouldn't dream of making changes without a good backup. ;)

Thanks for your help.

Rob

[PlainBill]

True, if you rely on MFS Tools to do the partitioning for you. But if you manually partition and/or coalesce the MFS partitions I haven't seen anything that would keep you from say having just one pair of MFS partitions on a large drive. Or, trying to more precisely match what originally existed on the Tivo drive. The possibilities are endless, I'm just trying to figure out if there are any benefits to one setup over another.
It is possible to create a TiVo drive from scratch; AlphaWolf has done it with his minimal 6.2 image. Hints on doing it are on this site, I don't have a link to them.

I do question if it's worth the effort. The last two minutes of a football game will still take 15 minutes; Dr. Phil will still be as boring. The menus will still be slow.

She'd kill me. :eek: But, being in the IT profession, I wouldn't dream of making changes without a good backup. ;)

Rob
OK, you've mastered the essential skill.

PlainBill

[joseph_goalie]

Hello,

I am running Fedora Core 5. I have 2 hd hooked up, /dev/hda is my os hd and /dev/hdb is the 30 GB hd from my Tivo 1: SVR-2000.

When I run pdisk -l /dev/hdb, it reports.

disk: No valid block 1 on '/dev/hdb'


I also tried running mfs_info, and I get the following

wrong magic 4d50 in partition 0

The Tivo hd works fine when I put it back in the tivo box. I also did a backup and restore as per hinsdale, and that seems to work fine.

What am I doing wrong? Is my tivo drive in a funky state, if so how do I get it to a normal state? Anyone?

Thanks

[eastwind]

You probably need to teach FC5 how to read a TiVo HD (tivopart) and for an S1 drive you'll need byteswapping turned on.

ew