Should boot CDs support LBA48? (split from: Common LBA48 mistakes)

[PTVupgrade]

Hello folks,

I've just been made aware of this thread and spend a good deal of time reading through it. Having a hard time understanding what the root of the problem is -- is it technical or is it expectations.

In advance, let me apologize of there is an 'expectation' issue that has arisen. The InstantCake CD for the 40-hour, 80-hour and 140-hour units, with the model number TCD240XXX is NOT lba48 aware because the kernel on these systems is also not lba48 aware. This is discussed on our own forums, but it is not highlighed as part of the product description for the main reason that the product is designed to work as such:

Boot the CD and use on any drive... it will install and expand the appropriate image and if the drive is > 137GB it will only use 137GB of it.

Now... I never envisioned InstantCake being used in any context other than creating a single or dual drive replacement set, and it looks like another applicatin has emerged, where folks want to layer additional things, including lba48 support on top.

Have I assessed the problem correctly, so far?

So again, if folks thought that this version of InstantCake was LBA48 aware, or the image would work on a larger drive, that is just not the case. It wouldn't be good to insert those changes into the product because a software update would render them unusable in the future and cause lots of problems, I'm sure you'll agree.

Now, is that the *only* problem or is there another problem that I'm not able to get my tiny little mind around? Are people now unable to layer the additional hacks on top of the newly imaged drive due to a shortcoming in InstantCake, or is this just an education issue?

I'm more than willing to consider making changes to the distribution if it doesn't create a problem for the more 'generic' customer, but if its just an issue of folks documenting how to use the existing product in a way other than it was originally intended, then I suspect those who are actually using it that way will have something more detailed to contribute.

Meanwhile, if there is a "bug" I need to know about it.

Thx

[chrised]

Are people now unable to layer the additional hacks on top of the newly imaged drive due to a shortcoming in InstantCake, or is this just an education issue?

I'm more than willing to consider making changes to the distribution if it doesn't create a problem for the more 'generic' customer, but if its just an issue of folks documenting how to use the existing product in a way other than it was originally intended, then I suspect those who are actually using it that way will have something more detailed to contribute.

Meanwhile, if there is a "bug" I need to know about it.

Thx

As someone who's used your product, and watched this thread, I consider this entirely a "newbie education issue". I disagree with ADH's warning on avoiding the product.

Based on the primary purpose of your product, clearly it is safest for you to leave it the way it is. The original kernel's don't support LBA48.

Now, if you want to make this easier for your customers using it for this application, I have a couple of suggestions.

1. Add a LBA48 boot option. Make sure you have a big disclaimer that this requires you to install a LBA48 kernel or you will suffer catestrophic results.

2. Add an option to restore WITHOUT expanding to full disk space. ie "Restore to original disk size only (User to run mfsadd later)".

With option 2, the idea is after restoring the image they can boot with a LBA48 MFStools and run mfsadd manually. Personally, I think this would be the best enhancement option, as option 1 is likely to cause some of your "normal" customers to use it wrong.

[alldeadhomiez]

Meanwhile, if there is a "bug" I need to know about it.

Selling a 2+yo kernel that doesn't support many/most of the hard drives sold today is clearly a bug. Please fix it. http://www.kernel.org

I really don't want to argue about politics, spin doctoring, blame, etc. in this support thread, so we can leave it at that. If you want to argue the point, PM me or start a new thread in the general discussions forum.

Edit: split it anyway, due to multiple replies.

Now, if you want to make this easier for your customers using it for this application, I have a couple of suggestions.

1. Add a LBA48 boot option. Make sure you have a big disclaimer that this requires you to install a LBA48 kernel or you will suffer catestrophic results.

2. Add an option to restore WITHOUT expanding to full disk space. ie "Restore to original disk size only (User to run mfsadd later)".

With option 2, the idea is after restoring the image they can boot with a LBA48 MFStools and run mfsadd manually. Personally, I think this would be the best enhancement option, as option 1 is likely to cause some of your "normal" customers to use it wrong.

I agree. The only correct way to handle this is to prompt the user to ascertain their intent.

Let's look at the alternative. Here are two bad ways to handle the situation:

1. Assume the user is installing an LBA48 kernel, so expand to 160GB on a 160GB drive. Don't warn them about it.

2. Install a malformed partition table on a >137GB drive, which cuts off the end of the drive. Don't warn them about it.

Whether the product is $20, $30, or free, there is no excuse for a product to do something unexpected without informing the user.

[PTVupgrade]

Selling a 2+yo kernel that doesn't support many/most of the hard drives sold today is clearly a bug. Please fix it. http://www.kernel.org

I really don't want to argue about politics, spin doctoring, blame, etc. in this support thread, so we can leave it at that. If you want to argue the point, PM me or start a new thread in the general discussions forum.

OK, so I don't want to fight about it either, so please don't pick one.

Back to the topic:

Until TiVo releases a kernel that addresses the problem, I don't think it would be smart to stick my lba48 kernel into the release. The problems that could cause would be enormous. Folks would unwittingly, or explicitly install that kernel, and then get the next release of the TiVo software that wouldn't have an lba48 kernel, and would then get hosed.

Its much better to have a layered product that installs an LBA48 kernel so that folks who know what they are doing, can do it, with the appropriate expectations.

Is it a limitation that 4.01 is not LBA48 aware, absolutely. Is my product defective because it doesn't change that, I don't agree with that.

[PTVupgrade]

Selling a 2+yo kernel that doesn't support many/most of the hard drives sold today is clearly a bug. Please fix it. http://www.kernel.org

I really don't want to argue about politics, spin doctoring, blame, etc. in this support thread, so we can leave it at that. If you want to argue the point, PM me or start a new thread in the general discussions forum.

Edit: split it anyway, due to multiple replies.



I agree. The only correct way to handle this is to prompt the user to ascertain their intent.

Let's look at the alternative. Here are two bad ways to handle the situation:

1. Assume the user is installing an LBA48 kernel, so expand to 160GB on a 160GB drive. Don't warn them about it.

2. Install a malformed partition table on a >137GB drive, which cuts off the end of the drive. Don't warn them about it.

Whether the product is $20, $30, or free, there is no excuse for a product to do something unexpected without informing the user.

k - looks like you edited your post. I agree with your last comment here. The challenge is anticipating what the user might do with it. The product works within the context that we've sold it in. Now the circumstances have changed, so its time to evolve it. That is fine. I just want to do it in a way that doesn't screw things up for those who have no idea what we are talking about.

What would you suggest?

[PTVupgrade]

As someone who's used your product, and watched this thread, I consider this entirely a "newbie education issue". I disagree with ADH's warning on avoiding the product.

Based on the primary purpose of your product, clearly it is safest for you to leave it the way it is. The original kernel's don't support LBA48.

Now, if you want to make this easier for your customers using it for this application, I have a couple of suggestions.

1. Add a LBA48 boot option. Make sure you have a big disclaimer that this requires you to install a LBA48 kernel or you will suffer catestrophic results.

2. Add an option to restore WITHOUT expanding to full disk space. ie "Restore to original disk size only (User to run mfsadd later)".

With option 2, the idea is after restoring the image they can boot with a LBA48 MFStools and run mfsadd manually. Personally, I think this would be the best enhancement option, as option 1 is likely to cause some of your "normal" customers to use it wrong.

Option 2 is what I would prefer. Obviously, its easier. It also prevents people from doing the wrong thing. My experience has been that people don't read disclaimers. Liability is not what I'm trying to avoid, mass confusion and broken TiVo systems IS what I'm trying to avoid.

Is anyone opposed to this?

[Jamie]

Option 2 is what I would prefer. Obviously, its easier. It also prevents people from doing the wrong thing. My experience has been that people don't read disclaimers. Liability is not what I'm trying to avoid, mass confusion and broken TiVo systems IS what I'm trying to avoid.
Seems to me that this option doesn't work unless you also switch to an lba48 kernel. If you create the intial partition table on a non lba48 system, a mfsadd later on a lba48 system will not fix it. This was discussed in the original thread this forked from:If you used a non-LBA48 kernel during either expansion or imaging, you will need to fix it by hand or start over.

[alldeadhomiez]

What would you suggest?

I would update the CD to use a recent kernel, then change the software to prompt the user in an ambiguous situation. I estimate that this would take a competent programmer under two hours.

[PTVupgrade]

I would update the CD to use a recent kernel, then change the software to prompt the user in an ambiguous situation. I estimate that this would take a competent programmer under two hours.

Well, it will definitely take longer than that in wall clock time, as running the business and doing development work are difficult things to multiplex. Testing will also be required.

Meanwhile, I'd politely suggest you reword your warning to be a bit more diplomatic, maybe something to warn folks that using it in a fashion other than what it was designed to do can result in nasty results. There is a fine line between warning folks and torting a business.

Lastly, its PTVupgrade not PTV upgrade; I have a trademark to defend, so please do not put the space in between the two words.

[Jamie]

Lastly, its PTVupgrade not PTV upgrade; I have a trademark to defend, so please do not put the space in between the two words.
The reason I've been adding the space is to foil the auto-linking substutition code that seems to be part of the forum software: see above, where I didn't add the space, and the quote comes out garbled.

[alldeadhomiez]

Meanwhile, I'd politely suggest you reword your warning to be a bit more diplomatic, maybe something to warn folks that using it in a fashion other than what it was designed to do can result in nasty results. There is a fine line between warning folks and torting a business.

Are you threatening me?

I'd politely suggest that you begin shipping products that don't write out malformed partition tables, so fewer people will have a reason to, ahem, "tort" you. :rolleyes:

[MuscleNerd]

There is a fine line between warning folks and torting a business.
PTV upgrade, your product is broken in many ways. It's a pathetic move to try to start stifling people for pointing that out.

And I'll continue to type "PTV upgrade", thank you very much.

Wow.

[PTVupgrade]

PTVupgrade, your product is broken in many ways. It's a pathetic move to try to start stifling people for pointing that out.

And I'll continue to type "PTVupgrade", thank you very much.

Wow.

Every single time I've attempted to be reasonable and approach a situation constructively, you guys ATTACK.

I'm sick of it.

And I've not attempted to stifle anyone. You guys want me gone. Tough. I'm not going away. I'd love to work together, but creating misconceptions and insulting me is not going to make life easier for anyone.

[MuscleNerd]

Every single time I've attempted to be reasonable and approach a situation constructively, you guys ATTACK.

I never even posted a message to you before, let alone attacked you. But it is increasingly obvious that you deserve the negative attention snowballing in your direction.

[JJBliss]

And I've not attempted to stifle anyone. You guys want me gone. Tough. I'm not going away. I'd love to work together, but creating misconceptions and insulting me is not going to make life easier for anyone.

Lou,

I am going to stay out of this one, since I am clearly a senior representative of this forum which you sponsor and the forum owner would not like to take an "official" stance that conflicts with a sponsor.

However, I need to just give you a heads up. Are you aware that the folks that you are writing to can very simply and without much effort crush your business and render your customers helpless and/or tivoless?

It does not make sense to make enemies of them. I hope you understand that they are as passionate as you are, and they are talented enough to cause you and your customers grief.

I have no indication that they are planning this, but I do warn you to look into the 25Xtreme/Tivoweb1.9.4 fiasco.

Hope this helps.

[PTVupgrade]

Lou,

I am going to stay out of this one, since I am clearly a senior representative of this forum which you sponsor. However, I need to just give you a heads up. Are you aware that the folks that you are writing to can very simply and without much effort crush your business and render your customers helpless and/or tivoless?

It does not make sense to make enemies of them. I hope you understand that they are as passionate as you are, and they are talented enough to cause you and your customers grief.

I have no indication that they are planning this, but I do warn you to look into the 25Xtreme/Tivoweb1.9.4 fiasco.

Hope this helps.

I am very aware of the fact that folks can make things very difficult for me (they already are) in many ways.

My point is that I have continually tried to NOT make enemies. In fact, I feel I've been diplomatic and constructive in attempts to do the right thing. In return, I am attacked, over and over again. I am not the only one with this perception.

Once again, I am trying to be reasonable; if that is not clear, then I apologize.

BUT there are some folks, no matter how hard I try, that just don't like me and what I do. I can't change that, as hard as I try.

Does that mean I'm making enemies? I don't know.

I'm just trying to make an appeal for a reasonable and constructive dialog. I have never threatened anyone around here, yet I continue to get pummelled. I have continually attempted to see things from the perspective of others, yet I am not getting the sense that my critics are looking at it from mine.

I don't want to make enemies here, its quite the opposite.

[alldeadhomiez]

I am going to stay out of this one, since I am clearly a senior representative of this forum which you sponsor and the forum owner would not like to take an "official" stance that conflicts with a sponsor.

On that note, I will also point out that my stance on the issue is purely from a technical/engineering perspective, and does not represent the opinion of DDB.

Every single time I've attempted to be reasonable and approach a situation constructively, you guys ATTACK.

I'm not sure what your computer background is, but people critique code all the time on technical mailing lists and forums. It seems like you take this sort of thing personally. Don't; it's not intended that way. Just fix the bugs, and things will blow over.

[MuscleNerd]

In fact, I feel I've been diplomatic and constructive....
I'm just trying to make an appeal for a reasonable and constructive dialog. I have never threatened anyone around here
You used the term "tort," which as I'm sure you know has legal connotations. If you truly attempt to "see things from the perspective of others," then you would realize that you were being inflammatory.

I rarely post about things not technically related to TiVo. So obviously, your attempt to be "reasonable" is not looking so reasonable to others.

[mcomer]

Hey folks, let's please work this out before it gets ugly (too late?). Having recently (just this month) used both the great information from this forum as well as the product from PTVupgrade to hack my three DTivos, I'd like to say that both are valuable. Given the PITA that finding good images can be, and the fact that Tivo has seemingly decided to permit PTVupgrade to offer them, I feel that they offer a good product at a reasonable price. To be honest, paying the small amount of money for the image from an approved source made me feel a lot better about my assertion that "hey, I'm not stealing anything from anyone" when it came time to hack my units. Besides, just starting up eMule anymore makes me feel like a target for the RIAA.

Clearly there are issues when the product is used by the uninformed for purposes which are arguably other than those intended. On the other hand, no product is without possibility of improvement, and it would be prudent to listen to the experts, particularly when their (the experts) work and expertise are the foundation of the product.

So many of the discussions on this board turn ugly. You guys do realize that while this place has the reputation as clearly THE source of accurate, complete technical information on Tivo's, it also has the reputation of being acrimonius and sometimes downright mean-spirited, right?

I haven't been here long enough to deserve an opinion, I know. I hope I'm not making myself flame-bait. :(

-Matt

[PTVupgrade]

You used the term "tort," which as I'm sure you know has legal connotations. If you truly attempt to "see things from the perspective of others," then you would realize that you were being inflammatory.

I rarely post about things not technically related to TiVo. So obviously, your attempt to be "reasonable" is not looking so reasonable to others.

OK, so let me try be the first to ratchet things down here, in the spirit of doing the right thing (I'll address ADH's most recent post here, too).

First of all, a big fat warning telling people there is a bug in my product is something that puts me on the defense. Had someone pointed out the shortcoming that exists, as a result of the product being used in a way that it was never intended to be used, it would have made much more sense to me.

So, yes, I used a strong word, but strong words were used prior to my use of them; my point is that it is a two way street. If you go back and look at many of my posts, even in response to inflammatory comments, I've attempted to be reasonable.

I am OK with the fact that not everyone agrees with me.

ADH - critiques are cool. Big fat warnings, are scary. I'm just trying to get to the root of the problem. I don't think the big fat warning was constructive as a critique should be. If there is a warning, all I'm asking is that it be qualified with a "why?" so that the distinction can be made. All I attempted to do this morning when I stepped in was find out what the problem is so I can try to fix it.

If there is a way to get this thread back on track so that I can get a better understanding of how I can modify the product so it meets people needs, that would be great; would just appreciate it if the inflammatory comments would stop coming my way as well, and I will behave. :-)

[alldeadhomiez]

Clearly there are issues when the product is used by the uninformed for purposes which are arguably other than those intended.

That's the baggage you get when you market a product to novice end-users. If you are selling the product, this is what you are getting paid for.

Again this is not a PTV issue at all. I have had plenty to say in the past about guides, all-in-one ISOs, pre-hacked units, and other such things. If you aim for a non-technical audience, you must support the product, you must keep it bug-free and up to date, and you must respect the contributors' wishes.

This is no small task, and definitely not something I would want to take on myself.

Big fat warnings, are scary. I'm just trying to get to the root of the problem. I don't think the big fat warning was constructive as a critique should be.

We have a 60+ post thread full of users who have beaten their head against the wall for the past week because of this bug. As InstantCake is recommended at the top of the "4.x on RID" files thread, it is only fair to warn users of its shortcomings so that others are not doomed to repeat the same mistake.

[PTVupgrade]

That's the baggage you get when you market a product to novice end-users. If you are selling the product, this is what you are getting paid for.

Again this is not a PTV issue at all. I have had plenty to say in the past about guides, all-in-one ISOs, pre-hacked units, and other such things. If you aim for a non-technical audience, you must support the product, you must keep it bug-free and up to date, and you must respect the contributors' wishes.

This is no small task, and definitely not something I would want to take on myself.



We have a 60+ post thread full of users who have beaten their head against the wall for the past week because of this bug. As InstantCake is recommended at the top of the "4.x on RID" files thread, it is only fair to warn users of its shortcomings so that others are not doomed to repeat the same mistake.

Fair enough.

Wish I had known about it. Would like to fix it. Ideally, I'd like an mfstools that was more flexible. Will try to do some workarounds with the scripts and warnings that will do the trick.

Thx.

[rc3105]

instantcake images (http://partners.ptvupgrade.com/z/40/CD20/) are still better than counting on p2p for a clean copy without other issues
*missing backgrounds, corrupted files, whatnot

my bad for overlooking that particular interaction :eek:


I'll revise the suggested install instructions and attempt to coordinate a fix with PTVupgrade as time is available

[newbie]

Prehaps you could just put the 4.X image on the http://downloads.ptvupgrade.com/Merchant2/merchant.mvc?Screen=PROD&Product_Code=LBA48-301DD&Category_Code= image file. People using a large hard drive won't be confused and expand the disk with your instant cake disk.

[malfunct]

Prehaps you could just put the 4.X image on the http://downloads.ptvupgrade.com/Merchant2/merchant.mvc?Screen=PROD&Product_Code=LBA48-301DD&Category_Code= image file. People using a large hard drive won't be confused and expand the disk with your instant cake disk.

I was thinking something similar. Why not have a "large" instantcake that supported lba48 and make it clear what its purpose is and the dangers it has lurking with it. With a bit of work and the tech that PTVupgrade (I hope I spelled that right) has licensed it could even install the image and then "fix" it so that when the system was booted things would still be ok. With a bit more futzing I think it could even go so far as to allow an upgrade by swapping kernels on the fly which then would at worst make the system fail to boot after upgrade because of kernel incompatibility, but you wouldn't fry the system by having the tivo come up without the lba48 support.

[PTVupgrade]

I was thinking something similar. Why not have a "large" instantcake that supported lba48 and make it clear what its purpose is and the dangers it has lurking with it. With a bit of work and the tech that PTVupgrade (I hope I spelled that right) has licensed it could even install the image and then "fix" it so that when the system was booted things would still be ok. With a bit more futzing I think it could even go so far as to allow an upgrade by swapping kernels on the fly which then would at worst make the system fail to boot after upgrade because of kernel incompatibility, but you wouldn't fry the system by having the tivo come up without the lba48 support.

Well...

I don't think it would be a good idea to release an InstantCake product directly on an LBA48 CD without and lba48 kernel that goes along with it. Even with an LBA48 kernel, it would have to be loaded with disclaimers because of the inevitable software upgrade that would occur with an lba48-less kernel along with it. I do understand why this would be useful to some, especially aroudn here, but it creates other problems for the "plain vanilla" types.

So how about this... and alternative IC build script that sits on an LBA48 boot CD; lets you pluck off your IC image from your "normal" IC distro; you can use the NST framework for the additional goodies, so that stuff stays separate. For good measure the IC distro contains warnings about its non-lba48 aware-ness; gives you the opportunity to restore without expanding but also gives you the ability to expand with a warning about doing so on drives > 137GB?

That can be done, as a start, at least.

[alldeadhomiez]

I don't think it would be a good idea to release an InstantCake product directly on an LBA48 CD without and lba48 kernel that goes along with it. Even with an LBA48 kernel, it would have to be loaded with disclaimers because of the inevitable software upgrade that would occur with an lba48-less kernel along with it. I do understand why this would be useful to some, especially aroudn here, but it creates other problems for the "plain vanilla" types.

This is not a difficult problem.

It would be trivial to boot a modern kernel, and expand to either the 137GB boundary or the full drive size after prompting the user (and explaining both choices). Obviously, if the drive is smaller than 137GB, the prompt will be skipped. In either case, a correct partition table would be written, so that expansion is possible later on.

[PTVupgrade]

This is not a difficult problem.

It would be trivial to boot a modern kernel, and expand to either the 137GB boundary or the full drive size after prompting the user (and explaining both choices). Obviously, if the drive is smaller than 137GB, the prompt will be skipped. In either case, a correct partition table would be written, so that expansion is possible later on.

That's ideally what I'd like to do; I have no idea how to tell MFStools only to go to the 137GB boundary.

[alldeadhomiez]

That's ideally what I'd like to do; I have no idea how to tell MFStools only to go to the 137GB boundary.

You edit the source code.

[PTVupgrade]

You edit the source code.

I will take a look. Unless its glaringly obvious (which I'm sure it is to you, and I don't mean that facetiously); I will have to script the workarounds here. There have been several shortcomings of MFStools that have plagued me, but C has never been my strongpoint. My programming background is porting and tuning 50,000 line FORTRAN apps for parallel supercomputers; a dated skill, at best.

[PTVupgrade]

Folks,

I've just uploaded a beta version of a new lba48 CD to my site with an additional script for testing, and a few other changes.

Based upon the recommendations of a few helpful folks, I've created a new version of the "PTVbake" script which is part of the standard InstantCake distribution.

What I was not aware of is a shortcoming in MFStools that causes problems when used to restore an image onto a drive > 137GB; even when used in conjunction with a non-lba48 CD.

This came as a shock to me. I was under the impression that best-practices dictated the following:

When restoring an LBA48-enabled TiVo image, and using a drive greater than 137GB, use an LBA48 boot CD. If the target drive is < 137GB, it does not matter.

When restoring a non-LBA48 TiVo image, and using a drive greater than 137GB, do not use an LBA48 boot CD, unless you are going to transplant a new kernel; instead, use a non-LBA48 boot CD, and the 137GB truncation that would ensue is expected and OK.

I am now finding out that this is not OK. I think the reality of the situation is that it is sort of OK under certain circumstances, but if transplants or expansions are to take place at a later date, there are going to be problems. And such appears to be the issue with folks who have been attempting to use InstantCake for Series2 non-lba48 units which are installed from a non-lba48 aware CD.

This workaround:

An LBA48 CD with an alternate PTVbake script allows you to easily install an image from any InstantCake CD and gives you a few more options during the installation process. Most importantly, the user is asked whether they want to 'expand' using traditional methods (the -x option) or whether they just want to do a generic restore with no expansion.

This *should* allow folks to get a clean and normal partition table and then do their expansions, kernel transplants, and other various and assorted things without a corrupted partition table from the start.

I hope that is the crux of the issue.

The second point I want to highlight is that when rebuilding this CD, I noticed something else which needed to be addressed. The kernel was not being loaded with byteswapping for hdb, hdc, and hdd. That means mfsinfo, mfsadd, and probably other things were unable to read the partition table -- I do not know whether this is related to the problem folks have been experiencing.

So, it is important to know that this version of the CD boots in "swap" mode and if you are accustomed to faster backups/restores, you can use the "noswap" command boot time. In either circumstance, the PTVbake-special script will/should run without incident.

OK, enough of this for now...

[edit: link removed... going gold with the new CD shortly...]

Going to "unbundle" the PTVbake-special script and post it here, as well as in its own thread with instructions, shortly.

To use the modified PTVbake script, in conjunction with an existing InstantCake CD, just boot with this CD (normal or noswap) and run,

PTVbake-special

once its booted (there are also supplemental instructions you can get by hitting "F6" at boot time.

Swap out the boot CD for your InstantCake CD at any time prior to the actual restore process. The CDROM volume will be automounted by the script right before restore-time and unmounted before it exits. If you've already manually mounted your CDROM drive already, the script will give you some ugly warnings, but will still work.

I hope this is useful; mods, if this is the wrong spot for this, please relocate or delete as you see fit. Meanwhile this download will only be available for a limited time, hopefully we can mold this into something useful, based upon any feedback.

Heading out for the evening, will check in tomorrow and hopefully get something finalized over the next few days.

Thx