Is there a file system that does error correction?











up vote
4
down vote

favorite












I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.



I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.



In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.



To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.



Is there something like this?










share|improve this question




















  • 1




    It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
    – David Schwartz
    Oct 21 '12 at 7:47












  • Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
    – Alfatau
    Oct 21 '12 at 8:02












  • If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
    – David Schwartz
    Oct 21 '12 at 8:03












  • I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
    – Alfatau
    Oct 21 '12 at 8:06






  • 1




    Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
    – Alfatau
    Oct 21 '12 at 8:13















up vote
4
down vote

favorite












I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.



I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.



In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.



To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.



Is there something like this?










share|improve this question




















  • 1




    It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
    – David Schwartz
    Oct 21 '12 at 7:47












  • Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
    – Alfatau
    Oct 21 '12 at 8:02












  • If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
    – David Schwartz
    Oct 21 '12 at 8:03












  • I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
    – Alfatau
    Oct 21 '12 at 8:06






  • 1




    Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
    – Alfatau
    Oct 21 '12 at 8:13













up vote
4
down vote

favorite









up vote
4
down vote

favorite











I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.



I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.



In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.



To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.



Is there something like this?










share|improve this question















I am looking for some virtual file system (e.g. with virtual hard disk) or some other kind of working solution to create archives or data backups with error correction redundancy.



I tried looking for something like Windows 7 VHD, which can be displayed as a new storage device with its own FS, but having a certain configurable percentile of recovery warranty on the files.



In other words, I'd like to have a VHD or something similar, which I can configure setting a certain recovery percentile (e.g. 10%) and the target is the files on this drive could be corrupted up to 10% and they will be recoverable.



To be more precise, the idea is similar to an "encrypted" FS (eg. TrueCrypt), but with redundancy for reliability instead of encryption for privacy and security.



Is there something like this?







backup filesystems file-management error-correction






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 23 at 1:28









phuclv

8,78563688




8,78563688










asked Oct 21 '12 at 7:32









Alfatau

264135




264135








  • 1




    It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
    – David Schwartz
    Oct 21 '12 at 7:47












  • Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
    – Alfatau
    Oct 21 '12 at 8:02












  • If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
    – David Schwartz
    Oct 21 '12 at 8:03












  • I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
    – Alfatau
    Oct 21 '12 at 8:06






  • 1




    Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
    – Alfatau
    Oct 21 '12 at 8:13














  • 1




    It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
    – David Schwartz
    Oct 21 '12 at 7:47












  • Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
    – Alfatau
    Oct 21 '12 at 8:02












  • If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
    – David Schwartz
    Oct 21 '12 at 8:03












  • I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
    – Alfatau
    Oct 21 '12 at 8:06






  • 1




    Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
    – Alfatau
    Oct 21 '12 at 8:13








1




1




It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47






It sounds like what you want is software RAID. There's no point in error correction on one drive because the drive as a whole is the most likely point of failure and drives already have error detection in hardware.
– David Schwartz
Oct 21 '12 at 7:47














Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02






Software RAID needs more than one hard drive. This would be a good solution if i could reserve a PC with multiple hard drives and a software raid layer over them. Really, this is a "bad copy" of a RAID hardware solution, so it's not what i was looking for. I need to plug an USB external storage to some notebooks and back up data. I know a single hard drive is not a "real" solution because with a mechanical failure of the drive all data will be lost.
– Alfatau
Oct 21 '12 at 8:02














If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03






If you only have one drive, it's hopeless. The most likely failure is the loss of that drive, and then what can you do? Also, something odd is going on with your question. If this is for backup, why does it need to be so reliable? Won't you still have the original if the backup system fails? If you really mean this is instead of backup, you're really going the wrong way!
– David Schwartz
Oct 21 '12 at 8:03














I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06




I simply want a "more reliable" solution then a standard one, because often happens that some few clusters damages over time. If data are not lost after the damaged clusters detection, i can replace the new drive having not lost any data.
– Alfatau
Oct 21 '12 at 8:06




1




1




Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13




Considering to use 2+ drives and replicate backups is in the same direction of disk redundancy, so it is an hw solution. Then i think it will be better to buy a NAS with raid support.
– Alfatau
Oct 21 '12 at 8:13










3 Answers
3






active

oldest

votes

















up vote
1
down vote













I haven't heard of one that works like a block-level filesystem.



Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.



You could always make your files redundant yourself using a PAR2 tool such as QuickPar.






share|improve this answer




























    up vote
    1
    down vote













    See https://www.thanassis.space/rsbep.html



    This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.



    To pitty there no filesystem which uses this technique..



    https://en.wikipedia.org/wiki/Reed–Solomon_error_correction






    share|improve this answer




























      up vote
      1
      down vote













      I'm not sure about virtual filesystem but there are many "real" file systems that do error correction






      • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem


      • Btrfs – A file system based on B-Trees, created by Oracle Corporation.


      • HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.


      • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.


      • Reliance – A transactional file system with CRCs, created by Datalight.


      • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.


      • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.


      • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)


      https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance




      Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.






      share|improve this answer





















        Your Answer








        StackExchange.ready(function() {
        var channelOptions = {
        tags: "".split(" "),
        id: "3"
        };
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function() {
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled) {
        StackExchange.using("snippets", function() {
        createEditor();
        });
        }
        else {
        createEditor();
        }
        });

        function createEditor() {
        StackExchange.prepareEditor({
        heartbeatType: 'answer',
        convertImagesToLinks: true,
        noModals: true,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: 10,
        bindNavPrevention: true,
        postfix: "",
        imageUploader: {
        brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
        contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
        allowUrls: true
        },
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        });


        }
        });














         

        draft saved


        draft discarded


















        StackExchange.ready(
        function () {
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f490527%2fis-there-a-file-system-that-does-error-correction%23new-answer', 'question_page');
        }
        );

        Post as a guest















        Required, but never shown

























        3 Answers
        3






        active

        oldest

        votes








        3 Answers
        3






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes








        up vote
        1
        down vote













        I haven't heard of one that works like a block-level filesystem.



        Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.



        You could always make your files redundant yourself using a PAR2 tool such as QuickPar.






        share|improve this answer

























          up vote
          1
          down vote













          I haven't heard of one that works like a block-level filesystem.



          Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.



          You could always make your files redundant yourself using a PAR2 tool such as QuickPar.






          share|improve this answer























            up vote
            1
            down vote










            up vote
            1
            down vote









            I haven't heard of one that works like a block-level filesystem.



            Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.



            You could always make your files redundant yourself using a PAR2 tool such as QuickPar.






            share|improve this answer












            I haven't heard of one that works like a block-level filesystem.



            Tahoe-LAFS does have this feature but it works more like FTP, it's slow, and really is designed to spread shares amongst multiple hosts.



            You could always make your files redundant yourself using a PAR2 tool such as QuickPar.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Jul 28 '13 at 14:27









            LawrenceC

            58.4k10100178




            58.4k10100178
























                up vote
                1
                down vote













                See https://www.thanassis.space/rsbep.html



                This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.



                To pitty there no filesystem which uses this technique..



                https://en.wikipedia.org/wiki/Reed–Solomon_error_correction






                share|improve this answer

























                  up vote
                  1
                  down vote













                  See https://www.thanassis.space/rsbep.html



                  This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.



                  To pitty there no filesystem which uses this technique..



                  https://en.wikipedia.org/wiki/Reed–Solomon_error_correction






                  share|improve this answer























                    up vote
                    1
                    down vote










                    up vote
                    1
                    down vote









                    See https://www.thanassis.space/rsbep.html



                    This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.



                    To pitty there no filesystem which uses this technique..



                    https://en.wikipedia.org/wiki/Reed–Solomon_error_correction






                    share|improve this answer












                    See https://www.thanassis.space/rsbep.html



                    This creates extra files with data for error correction using the Reed-Solomon error correction used on audio cd's in the past.



                    To pitty there no filesystem which uses this technique..



                    https://en.wikipedia.org/wiki/Reed–Solomon_error_correction







                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Feb 22 '17 at 20:01









                    Jeroen

                    111




                    111






















                        up vote
                        1
                        down vote













                        I'm not sure about virtual filesystem but there are many "real" file systems that do error correction






                        • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem


                        • Btrfs – A file system based on B-Trees, created by Oracle Corporation.


                        • HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.


                        • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.


                        • Reliance – A transactional file system with CRCs, created by Datalight.


                        • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.


                        • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.


                        • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)


                        https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance




                        Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.






                        share|improve this answer

























                          up vote
                          1
                          down vote













                          I'm not sure about virtual filesystem but there are many "real" file systems that do error correction






                          • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem


                          • Btrfs – A file system based on B-Trees, created by Oracle Corporation.


                          • HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.


                          • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.


                          • Reliance – A transactional file system with CRCs, created by Datalight.


                          • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.


                          • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.


                          • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)


                          https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance




                          Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.






                          share|improve this answer























                            up vote
                            1
                            down vote










                            up vote
                            1
                            down vote









                            I'm not sure about virtual filesystem but there are many "real" file systems that do error correction






                            • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem


                            • Btrfs – A file system based on B-Trees, created by Oracle Corporation.


                            • HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.


                            • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.


                            • Reliance – A transactional file system with CRCs, created by Datalight.


                            • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.


                            • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.


                            • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)


                            https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance




                            Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.






                            share|improve this answer












                            I'm not sure about virtual filesystem but there are many "real" file systems that do error correction






                            • Bcachefs - It's not yet upstream, full data and metadata checksumming, bcache is the bottom half of the filesystem


                            • Btrfs – A file system based on B-Trees, created by Oracle Corporation.


                            • HAMMER – DragonflyBSD's primary filesystem, created by Matt Dillon.


                            • ReFS (Resilient File System) – A file system by Microsoft with built-in resiliency features.


                            • Reliance – A transactional file system with CRCs, created by Datalight.


                            • Reliance Nitro – A tree-based transactional file system with CRCs, developed for high performance and reliability in embedded systems, from Datalight.


                            • NOVA – The "non-volatile memory accelerated" file system for persistent main memory.


                            • ZFS – Created by Sun Microsystems for use on Solaris 10 and OpenSolaris, ported to FreeBSD 7.0, NetBSD (as of 08/2009), Linux and to FUSE (not to be confused with the two zFSes from IBM)


                            https://en.wikipedia.org/wiki/List_of_file_systems#File_systems_with_built-in_fault-tolerance




                            Of course you can use those in a virtual hard drive and let them handle the error correction. There's no difference whether the file system driver reads from a physical or a virtual drive, since they only receive a stream of bytes.







                            share|improve this answer












                            share|improve this answer



                            share|improve this answer










                            answered Nov 22 at 15:44









                            phuclv

                            8,78563688




                            8,78563688






























                                 

                                draft saved


                                draft discarded



















































                                 


                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function () {
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f490527%2fis-there-a-file-system-that-does-error-correction%23new-answer', 'question_page');
                                }
                                );

                                Post as a guest















                                Required, but never shown





















































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown

































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown







                                Popular posts from this blog

                                flock() on closed filehandle LOCK_FILE at /usr/bin/apt-mirror

                                Mangá

                                Eduardo VII do Reino Unido