Is there a file system that does error correction?
up vote
4
down vote
favorite
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
|
show 4 more comments
up vote
4
down vote
favorite
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
|
show 4 more comments
up vote
4
down vote
favorite
up vote
4
down vote
favorite
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.
I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.
In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.
To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.
Is there something like this?
backup filesystems file-management error-correction
backup filesystems file-management error-correction
edited Nov 23 at 1:28
phuclv
8,78563688
8,78563688
asked Oct 21 '12 at 7:32
Alfatau
264135
264135
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
|
show 4 more comments
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
1
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13
|
show 4 more comments
3 Answers
3
active
oldest
votes
up vote
1
down vote
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
add a comment |
up vote
1
down vote
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
add a comment |
up vote
1
down vote
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
add a comment |
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
add a comment |
up vote
1
down vote
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
add a comment |
up vote
1
down vote
up vote
1
down vote
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
I haven't heard of one that works like a block-level filesystem.
Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.
You could always make your files redundant yourself using a PAR2 tool such as QuickPar.
answered Jul 28 '13 at 14:27
LawrenceC
58.4k10100178
58.4k10100178
add a comment |
add a comment |
up vote
1
down vote
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
add a comment |
up vote
1
down vote
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
add a comment |
up vote
1
down vote
up vote
1
down vote
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
See https://www.thanassis.space/rsbep.html
This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.
To pitty there no filesystem which uses this technique..
https://en.wikipedia.org/wiki/Reed–Solomon_error_correction
answered Feb 22 '17 at 20:01
Jeroen
111
111
add a comment |
add a comment |
up vote
1
down vote
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
add a comment |
up vote
1
down vote
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
add a comment |
up vote
1
down vote
up vote
1
down vote
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
I'm not sure about virtual filesystem but there are many "real" file systems that do error correction
Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem
Btrfs – A file system based on B-Trees, created by Oracle Corporation.
HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.
ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.
Reliance – A transactional file system with CRCs, created by Datalight.
Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.
NOVA – The "non-volatile memory accelerated" file system for persistent main memory.
ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)
https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance
Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.
answered Nov 22 at 15:44
phuclv
8,78563688
8,78563688
add a comment |
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f490527%2fis-there-a-file-system-that-does-error-correction%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47
Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02
If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03
I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06
1
Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13