Powershell Invoke-WebRequest with Method Head allocate too much RAM
up vote
0
down vote
favorite
Problem found on server with Powershell v 4.0
(server with Powershell 5.1 they do not seem affected by the same problem)
This a strange behavior of Invoke-WebRequest command.
Take this example:
For some reason, I need to know the file size before download it on my server.
Then I use the following commands:
$WebClient = Invoke-WebRequest -Uri $element -Method Head -Credential $Cred
$filesize = $webClient.Headers.'Content-Length'
The problem is before send $WebClient request Powershell use (for Commit RAM) about 120MB, but after the request powershell enlarge your RAM a the file $filesize value (example remote file is 800MB, new Commit RAM is 920MB).
When you work with Powershell WinRM session limited at 1GB of RAM this can be a problem. (OOM)
command-line memory powershell download oom
asked Dec 5 at 9:47
Max Monterumisi
1
add a comment |
up vote
0
down vote
favorite
Problem found on server with Powershell v 4.0
(server with Powershell 5.1 they do not seem affected by the same problem)
This a strange behavior of Invoke-WebRequest command.
Take this example:
For some reason, I need to know the file size before download it on my server.
Then I use the following commands:
$WebClient = Invoke-WebRequest -Uri $element -Method Head -Credential $Cred
$filesize = $webClient.Headers.'Content-Length'
The problem is before send $WebClient request Powershell use (for Commit RAM) about 120MB, but after the request powershell enlarge your RAM a the file $filesize value (example remote file is 800MB, new Commit RAM is 920MB).
When you work with Powershell WinRM session limited at 1GB of RAM this can be a problem. (OOM)
command-line memory powershell download oom
asked Dec 5 at 9:47
Max Monterumisi
1
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Problem found on server with Powershell v 4.0
(server with Powershell 5.1 they do not seem affected by the same problem)
This a strange behavior of Invoke-WebRequest command.
Take this example:
For some reason, I need to know the file size before download it on my server.
Then I use the following commands:
$WebClient = Invoke-WebRequest -Uri $element -Method Head -Credential $Cred
$filesize = $webClient.Headers.'Content-Length'
The problem is before send $WebClient request Powershell use (for Commit RAM) about 120MB, but after the request powershell enlarge your RAM a the file $filesize value (example remote file is 800MB, new Commit RAM is 920MB).
When you work with Powershell WinRM session limited at 1GB of RAM this can be a problem. (OOM)
command-line memory powershell download oom
asked Dec 5 at 9:47
Max Monterumisi
1
Problem found on server with Powershell v 4.0
(server with Powershell 5.1 they do not seem affected by the same problem)
This a strange behavior of Invoke-WebRequest command.
Take this example:
For some reason, I need to know the file size before download it on my server.
Then I use the following commands:
$WebClient = Invoke-WebRequest -Uri $element -Method Head -Credential $Cred
$filesize = $webClient.Headers.'Content-Length'
The problem is before send $WebClient request Powershell use (for Commit RAM) about 120MB, but after the request powershell enlarge your RAM a the file $filesize value (example remote file is 800MB, new Commit RAM is 920MB).
When you work with Powershell WinRM session limited at 1GB of RAM this can be a problem. (OOM)
command-line memory powershell download oom
command-line memory powershell download oom
asked Dec 5 at 9:47
Max Monterumisi
1
asked Dec 5 at 9:47
Max Monterumisi
1
asked Dec 5 at 9:47
Max Monterumisi
1
asked Dec 5 at 9:47
Max Monterumisi
1
asked Dec 5 at 9:47
Max Monterumisi
1
1
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
You can use Invoke-WebRequest with the HEAD method to get just the headers and not download anything. If the resource you're requesting has a known length, then you'll get a Content-Length header which you can use:
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'
Just note that not all servers return the Content-Length header for
all requests. In that case you will need to use your above method
which reads the entire file,
although it's slower and wasteful of memory.
You can also interface directly to the Windows DLLs that do Internet requests,
either
WinINet
(not available on Windows Server), or using
Windows HTTP Services.
answered Dec 5 at 11:18
harrymc
251k11259558
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
Actually you are right and there is a way usingInvoke-WebRequest. I rewrote my answer.
– harrymc
Dec 5 at 13:52
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1380954%2fpowershell-invoke-webrequest-with-method-head-allocate-too-much-ram%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
You can use Invoke-WebRequest with the HEAD method to get just the headers and not download anything. If the resource you're requesting has a known length, then you'll get a Content-Length header which you can use:
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'
Just note that not all servers return the Content-Length header for
all requests. In that case you will need to use your above method
which reads the entire file,
although it's slower and wasteful of memory.
You can also interface directly to the Windows DLLs that do Internet requests,
either
WinINet
(not available on Windows Server), or using
Windows HTTP Services.
answered Dec 5 at 11:18
harrymc
251k11259558
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
Actually you are right and there is a way usingInvoke-WebRequest. I rewrote my answer.
– harrymc
Dec 5 at 13:52
add a comment |
up vote
0
down vote
You can use Invoke-WebRequest with the HEAD method to get just the headers and not download anything. If the resource you're requesting has a known length, then you'll get a Content-Length header which you can use:
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'
Just note that not all servers return the Content-Length header for
all requests. In that case you will need to use your above method
which reads the entire file,
although it's slower and wasteful of memory.
You can also interface directly to the Windows DLLs that do Internet requests,
either
WinINet
(not available on Windows Server), or using
Windows HTTP Services.
answered Dec 5 at 11:18
harrymc
251k11259558
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
Actually you are right and there is a way usingInvoke-WebRequest. I rewrote my answer.
– harrymc
Dec 5 at 13:52
add a comment |
up vote
0
down vote
up vote
0
down vote
You can use Invoke-WebRequest with the HEAD method to get just the headers and not download anything. If the resource you're requesting has a known length, then you'll get a Content-Length header which you can use:
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'
Just note that not all servers return the Content-Length header for
all requests. In that case you will need to use your above method
which reads the entire file,
although it's slower and wasteful of memory.
You can also interface directly to the Windows DLLs that do Internet requests,
either
WinINet
(not available on Windows Server), or using
Windows HTTP Services.
answered Dec 5 at 11:18
harrymc
251k11259558
You can use Invoke-WebRequest with the HEAD method to get just the headers and not download anything. If the resource you're requesting has a known length, then you'll get a Content-Length header which you can use:
(Invoke-WebRequest $url -Method Head).Headers.'Content-Length'
Just note that not all servers return the Content-Length header for
all requests. In that case you will need to use your above method
which reads the entire file,
although it's slower and wasteful of memory.
You can also interface directly to the Windows DLLs that do Internet requests,
either
WinINet
(not available on Windows Server), or using
Windows HTTP Services.
answered Dec 5 at 11:18
harrymc
251k11259558
edited Dec 5 at 13:51
answered Dec 5 at 11:18
harrymc
251k11259558
answered Dec 5 at 11:18
harrymc
251k11259558
answered Dec 5 at 11:18
harrymc
251k11259558
251k11259558
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
Actually you are right and there is a way usingInvoke-WebRequest. I rewrote my answer.
– harrymc
Dec 5 at 13:52
add a comment |
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
Actually you are right and there is a way usingInvoke-WebRequest. I rewrote my answer.
– harrymc
Dec 5 at 13:52
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
If you send and HEAD request, and the remote system know the Content-Lenght of the resource request, Powershell (and all other language) not receive the file, but only a the HEAD with the Content-Lenght inside. So, I guess, it makes no sense to allocate as much memory as the file is large if I only do a head request.
– Max Monterumisi
Dec 5 at 13:40
Actually you are right and there is a way using
Invoke-WebRequest. I rewrote my answer.– harrymc
Dec 5 at 13:52
Actually you are right and there is a way using
Invoke-WebRequest. I rewrote my answer.– harrymc
Dec 5 at 13:52
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1380954%2fpowershell-invoke-webrequest-with-method-head-allocate-too-much-ram%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
