What marks data as deleted - the hard drive or the filesystem?












0














Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.



However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?










share|improve this question





























    0














    Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.



    However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?










    share|improve this question



























      0












      0








      0







      Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.



      However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?










      share|improve this question















      Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.



      However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?







      hard-drive ssd filesystems ntfs wipe






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Dec 23 '18 at 23:18







      Hashim

















      asked Dec 23 '18 at 23:13









      HashimHashim

      3,02963059




      3,02963059






















          2 Answers
          2






          active

          oldest

          votes


















          2














          HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.



          Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.



          Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.



          This separation of layers works very well with hard drives.



          For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.



          One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.



          This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").



          The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.



          When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).



          Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.






          share|improve this answer























          • This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
            – Hashim
            Dec 24 '18 at 17:36












          • File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
            – grawity
            Dec 24 '18 at 18:11





















          1














          The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.






          share|improve this answer





















            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "3"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1387235%2fwhat-marks-data-as-deleted-the-hard-drive-or-the-filesystem%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            2 Answers
            2






            active

            oldest

            votes








            2 Answers
            2






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            2














            HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.



            Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.



            Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.



            This separation of layers works very well with hard drives.



            For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.



            One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.



            This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").



            The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.



            When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).



            Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.






            share|improve this answer























            • This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
              – Hashim
              Dec 24 '18 at 17:36












            • File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
              – grawity
              Dec 24 '18 at 18:11


















            2














            HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.



            Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.



            Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.



            This separation of layers works very well with hard drives.



            For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.



            One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.



            This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").



            The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.



            When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).



            Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.






            share|improve this answer























            • This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
              – Hashim
              Dec 24 '18 at 17:36












            • File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
              – grawity
              Dec 24 '18 at 18:11
















            2












            2








            2






            HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.



            Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.



            Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.



            This separation of layers works very well with hard drives.



            For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.



            One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.



            This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").



            The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.



            When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).



            Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.






            share|improve this answer














            HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.



            Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.



            Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.



            This separation of layers works very well with hard drives.



            For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.



            One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.



            This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").



            The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.



            When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).



            Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Dec 24 '18 at 17:33









            Hashim

            3,02963059




            3,02963059










            answered Dec 24 '18 at 1:59









            Kamil MaciorowskiKamil Maciorowski

            24.7k155277




            24.7k155277












            • This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
              – Hashim
              Dec 24 '18 at 17:36












            • File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
              – grawity
              Dec 24 '18 at 18:11




















            • This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
              – Hashim
              Dec 24 '18 at 17:36












            • File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
              – grawity
              Dec 24 '18 at 18:11


















            This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
            – Hashim
            Dec 24 '18 at 17:36






            This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
            – Hashim
            Dec 24 '18 at 17:36














            File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
            – grawity
            Dec 24 '18 at 18:11






            File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
            – grawity
            Dec 24 '18 at 18:11















            1














            The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.






            share|improve this answer


























              1














              The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.






              share|improve this answer
























                1












                1








                1






                The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.






                share|improve this answer












                The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Dec 24 '18 at 0:56









                LMiller7LMiller7

                1,3861510




                1,3861510






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Super User!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.





                    Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                    Please pay close attention to the following guidance:


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1387235%2fwhat-marks-data-as-deleted-the-hard-drive-or-the-filesystem%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    flock() on closed filehandle LOCK_FILE at /usr/bin/apt-mirror

                    Mangá

                    Eduardo VII do Reino Unido