What marks data as deleted - the hard drive or the filesystem?
Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.
However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?
hard-drive ssd filesystems ntfs wipe
add a comment |
Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.
However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?
hard-drive ssd filesystems ntfs wipe
add a comment |
Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.
However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?
hard-drive ssd filesystems ntfs wipe
Anyone with a cursory understanding of data recovery and sanitisation as it relates to hard drives is aware of the concept that data "permanently" deleted from within an operating system is not deleted, but simply marked as such in the MFT, and that the data exists on the drive until the sector that it occupies on the hard drive is overwritten with new data.
However, I recently realised that I'm hazy on whether this process is a function of the hard drive itself, or the filesystem on it (NTFS more than likely, since we're talking about the MFT). If it's the former, how different is this process in mechanical and solid state drives? If the latter, do other filesystems (apart from NTFS) handle the deletion of data radically differently?
hard-drive ssd filesystems ntfs wipe
hard-drive ssd filesystems ntfs wipe
edited Dec 23 '18 at 23:18
Hashim
asked Dec 23 '18 at 23:13
HashimHashim
3,02963059
3,02963059
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.
Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.
Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.
This separation of layers works very well with hard drives.
For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.
One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.
This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").
The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.
When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).
Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
add a comment |
The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1387235%2fwhat-marks-data-as-deleted-the-hard-drive-or-the-filesystem%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.
Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.
Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.
This separation of layers works very well with hard drives.
For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.
One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.
This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").
The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.
When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).
Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
add a comment |
HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.
Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.
Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.
This separation of layers works very well with hard drives.
For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.
One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.
This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").
The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.
When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).
Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
add a comment |
HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.
Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.
Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.
This separation of layers works very well with hard drives.
For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.
One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.
This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").
The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.
When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).
Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.
HDD as abstraction layer doesn't need the concept of deleted data. It's a block device. You can either read or write to its block, a sector of your choice. Whatever you write to the sector, you can read it later as many times as you want. If you overwrite the sector then consecutive read attempts will return the new data. There is no special state that indicates "no data". If the sector size is 512 bytes and the disk is healthy, then you will get 512 bytes from it every time you read. Some higher abstraction layer needs to know whether these bytes mean something or not.
Usually a filesystem is such a layer. The same byte sequence returned from the same sector may be a part of some file or empty space, depending on contents of other sectors (in your example: where MFT is stored). The term "empty space" here is misleading; "expendable bytes" would be better.
Since any sector has to return some data anyway when queried, it's a sane and common approach to leave the data intact while deleting a file and modify just the metadata (e.g. MFT) which tracks meaningful and expendable sectors. Common filesystems handle deletions this way.
This separation of layers works very well with hard drives.
For SSDs (flash memory in general) there are at least two reasons you'd like to let the device itself (i.e. its firmware controller) know which sectors are expendable.
One is the fact that you cannot directly overwrite data stored in flash memory cells (frankly I'm not sure if this applies to all technologies, but it applies at least to some of them and they are common). You need to erase cells, and then write new data to them during a separate operation.
This means that internally there is a state indicating "deleted" or "no data". If the device knows which sectors are expendable, it can erase them beforehand and when the time comes to write something, there is no delay. (Well, it's even more complicated. You can erase no less than a whole bunch of cells, a lot more than one 512-byte sector. For more information research "flash blocks and pages", "flash garbage collection").
The other reason is wear-leveling. A flash memory cell wears a little with every write (or rather with every erase). If you keep overwriting the same logical sector again and again, your modern SSD will remap it to different physical cells so they use up evenly. The device probably has a significant pool of "spare" cells it can use even if none of the logical sectors is known to be expendable, but the whole trick works better if the controller knows which sectors contain data that is no longer needed, so it can freely reuse cells mapped to them.
When your operating system (that understands the filesystem) informs your SSD which sectors are expendable, we call this trimming (see this answer of mine for some details).
Note that even if the whole SSD is internally erased, any sector will return some binary data when read (I guess the firmware will probably return all zeros, I haven't tested though); you cannot tell erased cells from cells actually containing this particular data just by reading from the device. Externally this mimics the traditional HDD behavior where there is no concept of erased (deleted) data at all.
edited Dec 24 '18 at 17:33
Hashim
3,02963059
3,02963059
answered Dec 24 '18 at 1:59
Kamil MaciorowskiKamil Maciorowski
24.7k155277
24.7k155277
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
add a comment |
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
This is a good, comprehensive answer, but it only comprehensively answers the first part of the question. I'd like to see more information about the filesystems that do or do not work this way; it seems highly relevant for data wiping to know whether the filesystem itself actually deletes in the way data wipers assumes it does.
– Hashim
Dec 24 '18 at 17:36
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
File wipers specifically do not assume the FS deletes data this or another way. Instead they overwrite a file's contents while it is still in normal (not deleted) state. So they're based on assumptions about the opposite thing: how the filesystem overwrites data still in use, and are unaffected by what happens on deletion.
– grawity
Dec 24 '18 at 18:11
add a comment |
The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.
add a comment |
The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.
add a comment |
The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.
The file system, for which the hard drive provides physical storage, would mark the file as deleted. The drive itself, conventional or SSD has no knowledge of files, folders, file systems, partitions, etc. These are all high level concepts implemented by the file system and other system software. The drive only understands blocks of data and has no understanding of what the data might be used for.
answered Dec 24 '18 at 0:56
LMiller7LMiller7
1,3861510
1,3861510
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1387235%2fwhat-marks-data-as-deleted-the-hard-drive-or-the-filesystem%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown