login to website with curl and download webpage
i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...
anyway, my code is as follows:
curl -k https://url.retail.publishedprices.co.il/login > 01.txt
sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt
set /p csrf=<02.txt
curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt
what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.
the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.
any help would be appreciated!
windows website authentication curl
add a comment |
i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...
anyway, my code is as follows:
curl -k https://url.retail.publishedprices.co.il/login > 01.txt
sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt
set /p csrf=<02.txt
curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt
what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.
the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.
any help would be appreciated!
windows website authentication curl
What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?
– CraftyB
Jan 28 at 14:56
windows 10 LTSB/C. a solution using powershell is fine. thank you.
– John
Jan 28 at 18:03
add a comment |
i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...
anyway, my code is as follows:
curl -k https://url.retail.publishedprices.co.il/login > 01.txt
sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt
set /p csrf=<02.txt
curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt
what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.
the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.
any help would be appreciated!
windows website authentication curl
i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...
anyway, my code is as follows:
curl -k https://url.retail.publishedprices.co.il/login > 01.txt
sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt
set /p csrf=<02.txt
curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt
what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.
the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.
any help would be appreciated!
windows website authentication curl
windows website authentication curl
asked Jan 28 at 12:59
JohnJohn
131117
131117
What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?
– CraftyB
Jan 28 at 14:56
windows 10 LTSB/C. a solution using powershell is fine. thank you.
– John
Jan 28 at 18:03
add a comment |
What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?
– CraftyB
Jan 28 at 14:56
windows 10 LTSB/C. a solution using powershell is fine. thank you.
– John
Jan 28 at 18:03
What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?
– CraftyB
Jan 28 at 14:56
What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?
– CraftyB
Jan 28 at 14:56
windows 10 LTSB/C. a solution using powershell is fine. thank you.
– John
Jan 28 at 18:03
windows 10 LTSB/C. a solution using powershell is fine. thank you.
– John
Jan 28 at 18:03
add a comment |
1 Answer
1
active
oldest
votes
As I was working on something similar at the time I quickly put this together for you:
Please change the path and filename to your requirements, make sure that the path has a trailing slash.
$path = "c:test"
$filename = "website.txt"
$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"
$ie = New-Object -com InternetExplorer.Application
$ie.visible = $true
$ie.silent = $false
$ie.navigate("$url")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}
$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()
start-sleep 5
$ie.navigate($url2)
start-sleep 10
$ie.Document.body.outerHTML | Out-File -FilePath $path$filename
This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.
After you have ran the script and are happy with the outcome you can then do the following.
Change the following so that IE does not open up on the desktop:
$ie.visible = $true
to
$ie.visible = $false
and add the following at the end of the script to ensure IE closes at the end:
$ie.quit()
Hope this helps.
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1399245%2flogin-to-website-with-curl-and-download-webpage%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
As I was working on something similar at the time I quickly put this together for you:
Please change the path and filename to your requirements, make sure that the path has a trailing slash.
$path = "c:test"
$filename = "website.txt"
$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"
$ie = New-Object -com InternetExplorer.Application
$ie.visible = $true
$ie.silent = $false
$ie.navigate("$url")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}
$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()
start-sleep 5
$ie.navigate($url2)
start-sleep 10
$ie.Document.body.outerHTML | Out-File -FilePath $path$filename
This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.
After you have ran the script and are happy with the outcome you can then do the following.
Change the following so that IE does not open up on the desktop:
$ie.visible = $true
to
$ie.visible = $false
and add the following at the end of the script to ensure IE closes at the end:
$ie.quit()
Hope this helps.
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
add a comment |
As I was working on something similar at the time I quickly put this together for you:
Please change the path and filename to your requirements, make sure that the path has a trailing slash.
$path = "c:test"
$filename = "website.txt"
$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"
$ie = New-Object -com InternetExplorer.Application
$ie.visible = $true
$ie.silent = $false
$ie.navigate("$url")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}
$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()
start-sleep 5
$ie.navigate($url2)
start-sleep 10
$ie.Document.body.outerHTML | Out-File -FilePath $path$filename
This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.
After you have ran the script and are happy with the outcome you can then do the following.
Change the following so that IE does not open up on the desktop:
$ie.visible = $true
to
$ie.visible = $false
and add the following at the end of the script to ensure IE closes at the end:
$ie.quit()
Hope this helps.
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
add a comment |
As I was working on something similar at the time I quickly put this together for you:
Please change the path and filename to your requirements, make sure that the path has a trailing slash.
$path = "c:test"
$filename = "website.txt"
$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"
$ie = New-Object -com InternetExplorer.Application
$ie.visible = $true
$ie.silent = $false
$ie.navigate("$url")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}
$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()
start-sleep 5
$ie.navigate($url2)
start-sleep 10
$ie.Document.body.outerHTML | Out-File -FilePath $path$filename
This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.
After you have ran the script and are happy with the outcome you can then do the following.
Change the following so that IE does not open up on the desktop:
$ie.visible = $true
to
$ie.visible = $false
and add the following at the end of the script to ensure IE closes at the end:
$ie.quit()
Hope this helps.
As I was working on something similar at the time I quickly put this together for you:
Please change the path and filename to your requirements, make sure that the path has a trailing slash.
$path = "c:test"
$filename = "website.txt"
$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"
$ie = New-Object -com InternetExplorer.Application
$ie.visible = $true
$ie.silent = $false
$ie.navigate("$url")
while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}
$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()
start-sleep 5
$ie.navigate($url2)
start-sleep 10
$ie.Document.body.outerHTML | Out-File -FilePath $path$filename
This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.
After you have ran the script and are happy with the outcome you can then do the following.
Change the following so that IE does not open up on the desktop:
$ie.visible = $true
to
$ie.visible = $false
and add the following at the end of the script to ensure IE closes at the end:
$ie.quit()
Hope this helps.
edited Jan 28 at 18:35
answered Jan 28 at 18:26
CraftyBCraftyB
1,12429
1,12429
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
add a comment |
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
... and how do you run it? ;-)
– Hannu
Jan 28 at 19:17
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1399245%2flogin-to-website-with-curl-and-download-webpage%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?
– CraftyB
Jan 28 at 14:56
windows 10 LTSB/C. a solution using powershell is fine. thank you.
– John
Jan 28 at 18:03