Why is file transfer speed decreased when copying multiple smaller files?











up vote
3
down vote

favorite












Let's say I'm copying a portable apps directory from PC to an external HDD.
An application.exe file is around 4 megabytes in size, but most other files are only tens of kilobytes in size, and most of those files take longer to copy/move than that bigger file having 4 megabytes.



Why is this?










share|improve this question


























    up vote
    3
    down vote

    favorite












    Let's say I'm copying a portable apps directory from PC to an external HDD.
    An application.exe file is around 4 megabytes in size, but most other files are only tens of kilobytes in size, and most of those files take longer to copy/move than that bigger file having 4 megabytes.



    Why is this?










    share|improve this question
























      up vote
      3
      down vote

      favorite









      up vote
      3
      down vote

      favorite











      Let's say I'm copying a portable apps directory from PC to an external HDD.
      An application.exe file is around 4 megabytes in size, but most other files are only tens of kilobytes in size, and most of those files take longer to copy/move than that bigger file having 4 megabytes.



      Why is this?










      share|improve this question













      Let's say I'm copying a portable apps directory from PC to an external HDD.
      An application.exe file is around 4 megabytes in size, but most other files are only tens of kilobytes in size, and most of those files take longer to copy/move than that bigger file having 4 megabytes.



      Why is this?







      file-transfer speed






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Dec 27 '14 at 1:06









      Ivan Ivković

      1331113




      1331113






















          4 Answers
          4






          active

          oldest

          votes

















          up vote
          6
          down vote













          When transferring data, if you move one large file, you only have to write the directory entry once and then stream the data and write more or less continuously. When you have many files, you write the directory info, move to the file area and write the data, then go back and write the directory entry for the next file and then move back to the data area and write the data and back and forth. The overhead of moving back and forth adds up.






          share|improve this answer




























            up vote
            5
            down vote













            If you transfer a file, theres an open+read+close at the source, and an open+write+close at the destination, with all the associated attributes (ctime/atime/mtime etc) being created at the destination. Fragmentation (higher seek times) will also affect the datarate. Congestion of e.g. a sata-channel may also be an issue (use a separate ports for each disk, buy a sata card if you run out). Bigger files have longer consecutive reads or writes, and will transfer faster. Im sure there's much more to it. Does this help you?






            share|improve this answer




























              up vote
              2
              down vote













              That's the way windows works, and that's the part HDD are bad at. Usually SSDs are good at copying small files. You can check out benchmarking of HDDs and SSDs about file transfer speeds. But you have to know that other things are in place, like windows is reading first the files, then your antivirus checks them, and then they get written to the new destination folder. Not to mention that when you cut a file, the deletion process is added. So, more files, more "real" time added to copy a bigger number of files.






              share|improve this answer




























                up vote
                -1
                down vote














                Think of it like this,



                The computer is assigned a task to take 10 books (10 files) from point A to point B, If they are individual then the computer needs to get one take it to point B and then come back to point A to pick up the next book and so on. Whereas if it needed to take one big book that was the equivalent weight of the 10 books combined it would take longer to move the book but it will be a quicker process because the computer only makes one trip.






                share|improve this answer



















                • 1




                  Nice metaphor, but I can't see why the two processes are similar.
                  – Máté Juhász
                  Oct 5 at 4:01











                Your Answer








                StackExchange.ready(function() {
                var channelOptions = {
                tags: "".split(" "),
                id: "3"
                };
                initTagRenderer("".split(" "), "".split(" "), channelOptions);

                StackExchange.using("externalEditor", function() {
                // Have to fire editor after snippets, if snippets enabled
                if (StackExchange.settings.snippets.snippetsEnabled) {
                StackExchange.using("snippets", function() {
                createEditor();
                });
                }
                else {
                createEditor();
                }
                });

                function createEditor() {
                StackExchange.prepareEditor({
                heartbeatType: 'answer',
                convertImagesToLinks: true,
                noModals: true,
                showLowRepImageUploadWarning: true,
                reputationToPostImages: 10,
                bindNavPrevention: true,
                postfix: "",
                imageUploader: {
                brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
                contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
                allowUrls: true
                },
                onDemand: true,
                discardSelector: ".discard-answer"
                ,immediatelyShowMarkdownHelp:true
                });


                }
                });














                draft saved

                draft discarded


















                StackExchange.ready(
                function () {
                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f857243%2fwhy-is-file-transfer-speed-decreased-when-copying-multiple-smaller-files%23new-answer', 'question_page');
                }
                );

                Post as a guest















                Required, but never shown

























                4 Answers
                4






                active

                oldest

                votes








                4 Answers
                4






                active

                oldest

                votes









                active

                oldest

                votes






                active

                oldest

                votes








                up vote
                6
                down vote













                When transferring data, if you move one large file, you only have to write the directory entry once and then stream the data and write more or less continuously. When you have many files, you write the directory info, move to the file area and write the data, then go back and write the directory entry for the next file and then move back to the data area and write the data and back and forth. The overhead of moving back and forth adds up.






                share|improve this answer

























                  up vote
                  6
                  down vote













                  When transferring data, if you move one large file, you only have to write the directory entry once and then stream the data and write more or less continuously. When you have many files, you write the directory info, move to the file area and write the data, then go back and write the directory entry for the next file and then move back to the data area and write the data and back and forth. The overhead of moving back and forth adds up.






                  share|improve this answer























                    up vote
                    6
                    down vote










                    up vote
                    6
                    down vote









                    When transferring data, if you move one large file, you only have to write the directory entry once and then stream the data and write more or less continuously. When you have many files, you write the directory info, move to the file area and write the data, then go back and write the directory entry for the next file and then move back to the data area and write the data and back and forth. The overhead of moving back and forth adds up.






                    share|improve this answer












                    When transferring data, if you move one large file, you only have to write the directory entry once and then stream the data and write more or less continuously. When you have many files, you write the directory info, move to the file area and write the data, then go back and write the directory entry for the next file and then move back to the data area and write the data and back and forth. The overhead of moving back and forth adds up.







                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Dec 27 '14 at 1:21









                    Blackbeagle

                    6,3611623




                    6,3611623
























                        up vote
                        5
                        down vote













                        If you transfer a file, theres an open+read+close at the source, and an open+write+close at the destination, with all the associated attributes (ctime/atime/mtime etc) being created at the destination. Fragmentation (higher seek times) will also affect the datarate. Congestion of e.g. a sata-channel may also be an issue (use a separate ports for each disk, buy a sata card if you run out). Bigger files have longer consecutive reads or writes, and will transfer faster. Im sure there's much more to it. Does this help you?






                        share|improve this answer

























                          up vote
                          5
                          down vote













                          If you transfer a file, theres an open+read+close at the source, and an open+write+close at the destination, with all the associated attributes (ctime/atime/mtime etc) being created at the destination. Fragmentation (higher seek times) will also affect the datarate. Congestion of e.g. a sata-channel may also be an issue (use a separate ports for each disk, buy a sata card if you run out). Bigger files have longer consecutive reads or writes, and will transfer faster. Im sure there's much more to it. Does this help you?






                          share|improve this answer























                            up vote
                            5
                            down vote










                            up vote
                            5
                            down vote









                            If you transfer a file, theres an open+read+close at the source, and an open+write+close at the destination, with all the associated attributes (ctime/atime/mtime etc) being created at the destination. Fragmentation (higher seek times) will also affect the datarate. Congestion of e.g. a sata-channel may also be an issue (use a separate ports for each disk, buy a sata card if you run out). Bigger files have longer consecutive reads or writes, and will transfer faster. Im sure there's much more to it. Does this help you?






                            share|improve this answer












                            If you transfer a file, theres an open+read+close at the source, and an open+write+close at the destination, with all the associated attributes (ctime/atime/mtime etc) being created at the destination. Fragmentation (higher seek times) will also affect the datarate. Congestion of e.g. a sata-channel may also be an issue (use a separate ports for each disk, buy a sata card if you run out). Bigger files have longer consecutive reads or writes, and will transfer faster. Im sure there's much more to it. Does this help you?







                            share|improve this answer












                            share|improve this answer



                            share|improve this answer










                            answered Dec 27 '14 at 1:19









                            user400344

                            31614




                            31614






















                                up vote
                                2
                                down vote













                                That's the way windows works, and that's the part HDD are bad at. Usually SSDs are good at copying small files. You can check out benchmarking of HDDs and SSDs about file transfer speeds. But you have to know that other things are in place, like windows is reading first the files, then your antivirus checks them, and then they get written to the new destination folder. Not to mention that when you cut a file, the deletion process is added. So, more files, more "real" time added to copy a bigger number of files.






                                share|improve this answer

























                                  up vote
                                  2
                                  down vote













                                  That's the way windows works, and that's the part HDD are bad at. Usually SSDs are good at copying small files. You can check out benchmarking of HDDs and SSDs about file transfer speeds. But you have to know that other things are in place, like windows is reading first the files, then your antivirus checks them, and then they get written to the new destination folder. Not to mention that when you cut a file, the deletion process is added. So, more files, more "real" time added to copy a bigger number of files.






                                  share|improve this answer























                                    up vote
                                    2
                                    down vote










                                    up vote
                                    2
                                    down vote









                                    That's the way windows works, and that's the part HDD are bad at. Usually SSDs are good at copying small files. You can check out benchmarking of HDDs and SSDs about file transfer speeds. But you have to know that other things are in place, like windows is reading first the files, then your antivirus checks them, and then they get written to the new destination folder. Not to mention that when you cut a file, the deletion process is added. So, more files, more "real" time added to copy a bigger number of files.






                                    share|improve this answer












                                    That's the way windows works, and that's the part HDD are bad at. Usually SSDs are good at copying small files. You can check out benchmarking of HDDs and SSDs about file transfer speeds. But you have to know that other things are in place, like windows is reading first the files, then your antivirus checks them, and then they get written to the new destination folder. Not to mention that when you cut a file, the deletion process is added. So, more files, more "real" time added to copy a bigger number of files.







                                    share|improve this answer












                                    share|improve this answer



                                    share|improve this answer










                                    answered Dec 27 '14 at 1:27









                                    Erlis D.

                                    708413




                                    708413






















                                        up vote
                                        -1
                                        down vote














                                        Think of it like this,



                                        The computer is assigned a task to take 10 books (10 files) from point A to point B, If they are individual then the computer needs to get one take it to point B and then come back to point A to pick up the next book and so on. Whereas if it needed to take one big book that was the equivalent weight of the 10 books combined it would take longer to move the book but it will be a quicker process because the computer only makes one trip.






                                        share|improve this answer



















                                        • 1




                                          Nice metaphor, but I can't see why the two processes are similar.
                                          – Máté Juhász
                                          Oct 5 at 4:01















                                        up vote
                                        -1
                                        down vote














                                        Think of it like this,



                                        The computer is assigned a task to take 10 books (10 files) from point A to point B, If they are individual then the computer needs to get one take it to point B and then come back to point A to pick up the next book and so on. Whereas if it needed to take one big book that was the equivalent weight of the 10 books combined it would take longer to move the book but it will be a quicker process because the computer only makes one trip.






                                        share|improve this answer



















                                        • 1




                                          Nice metaphor, but I can't see why the two processes are similar.
                                          – Máté Juhász
                                          Oct 5 at 4:01













                                        up vote
                                        -1
                                        down vote










                                        up vote
                                        -1
                                        down vote










                                        Think of it like this,



                                        The computer is assigned a task to take 10 books (10 files) from point A to point B, If they are individual then the computer needs to get one take it to point B and then come back to point A to pick up the next book and so on. Whereas if it needed to take one big book that was the equivalent weight of the 10 books combined it would take longer to move the book but it will be a quicker process because the computer only makes one trip.






                                        share|improve this answer















                                        Think of it like this,



                                        The computer is assigned a task to take 10 books (10 files) from point A to point B, If they are individual then the computer needs to get one take it to point B and then come back to point A to pick up the next book and so on. Whereas if it needed to take one big book that was the equivalent weight of the 10 books combined it would take longer to move the book but it will be a quicker process because the computer only makes one trip.







                                        share|improve this answer














                                        share|improve this answer



                                        share|improve this answer








                                        edited Oct 5 at 19:50

























                                        answered Oct 5 at 3:50









                                        TheLifeHacker

                                        11




                                        11








                                        • 1




                                          Nice metaphor, but I can't see why the two processes are similar.
                                          – Máté Juhász
                                          Oct 5 at 4:01














                                        • 1




                                          Nice metaphor, but I can't see why the two processes are similar.
                                          – Máté Juhász
                                          Oct 5 at 4:01








                                        1




                                        1




                                        Nice metaphor, but I can't see why the two processes are similar.
                                        – Máté Juhász
                                        Oct 5 at 4:01




                                        Nice metaphor, but I can't see why the two processes are similar.
                                        – Máté Juhász
                                        Oct 5 at 4:01


















                                        draft saved

                                        draft discarded




















































                                        Thanks for contributing an answer to Super User!


                                        • Please be sure to answer the question. Provide details and share your research!

                                        But avoid



                                        • Asking for help, clarification, or responding to other answers.

                                        • Making statements based on opinion; back them up with references or personal experience.


                                        To learn more, see our tips on writing great answers.





                                        Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                                        Please pay close attention to the following guidance:


                                        • Please be sure to answer the question. Provide details and share your research!

                                        But avoid



                                        • Asking for help, clarification, or responding to other answers.

                                        • Making statements based on opinion; back them up with references or personal experience.


                                        To learn more, see our tips on writing great answers.




                                        draft saved


                                        draft discarded














                                        StackExchange.ready(
                                        function () {
                                        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f857243%2fwhy-is-file-transfer-speed-decreased-when-copying-multiple-smaller-files%23new-answer', 'question_page');
                                        }
                                        );

                                        Post as a guest















                                        Required, but never shown





















































                                        Required, but never shown














                                        Required, but never shown












                                        Required, but never shown







                                        Required, but never shown

































                                        Required, but never shown














                                        Required, but never shown












                                        Required, but never shown







                                        Required, but never shown







                                        Popular posts from this blog

                                        Mouse cursor on multiple screens with different PPI

                                        Agildo Ribeiro

                                        Sometime when accessing a menu: “Ubuntu 16.04 has experienced an internal error”