login to website with curl and download webpage












0















i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...



anyway, my code is as follows:



curl -k https://url.retail.publishedprices.co.il/login > 01.txt

sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt

set /p csrf=<02.txt

curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt


what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.



the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.



any help would be appreciated!










share|improve this question























  • What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?

    – CraftyB
    Jan 28 at 14:56











  • windows 10 LTSB/C. a solution using powershell is fine. thank you.

    – John
    Jan 28 at 18:03
















0















i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...



anyway, my code is as follows:



curl -k https://url.retail.publishedprices.co.il/login > 01.txt

sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt

set /p csrf=<02.txt

curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt


what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.



the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.



any help would be appreciated!










share|improve this question























  • What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?

    – CraftyB
    Jan 28 at 14:56











  • windows 10 LTSB/C. a solution using powershell is fine. thank you.

    – John
    Jan 28 at 18:03














0












0








0








i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...



anyway, my code is as follows:



curl -k https://url.retail.publishedprices.co.il/login > 01.txt

sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt

set /p csrf=<02.txt

curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt


what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.



the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.



any help would be appreciated!










share|improve this question














i am trying to log-in into a website and download a webpage.
do not be alarmed- the website and credentials are public knowledge...



anyway, my code is as follows:



curl -k https://url.retail.publishedprices.co.il/login > 01.txt

sfk xed 01.txt "_[start][0.10000 bytes]csrftoken__" +xed "_/>**__" +xed "_q id=qcsrftokenq value=q__" +xed "_q __" -tofile 02.txt

set /p csrf=<02.txt

curl -k -d "username=retalix&password=12345&csrftoken=%csrf%" https://url.retail.publishedprices.co.il/file/d/RamiLevi/ > RL.txt


what the code does is download the login page, extract the csrf variable and use it, plus the username/password, to log-in another sub-"domain" in the same website.



the end result should be a webpage that is saved into "RL.txt" but none of the solutions i could find worked and i can't understand why.



any help would be appreciated!







windows website authentication curl






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Jan 28 at 12:59









JohnJohn

131117




131117













  • What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?

    – CraftyB
    Jan 28 at 14:56











  • windows 10 LTSB/C. a solution using powershell is fine. thank you.

    – John
    Jan 28 at 18:03



















  • What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?

    – CraftyB
    Jan 28 at 14:56











  • windows 10 LTSB/C. a solution using powershell is fine. thank you.

    – John
    Jan 28 at 18:03

















What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?

– CraftyB
Jan 28 at 14:56





What version of windows are you using?, are you restricted to using curl/sfk? Would you consider powershell?

– CraftyB
Jan 28 at 14:56













windows 10 LTSB/C. a solution using powershell is fine. thank you.

– John
Jan 28 at 18:03





windows 10 LTSB/C. a solution using powershell is fine. thank you.

– John
Jan 28 at 18:03










1 Answer
1






active

oldest

votes


















1














As I was working on something similar at the time I quickly put this together for you:



Please change the path and filename to your requirements, make sure that the path has a trailing slash.



$path = "c:test"
$filename = "website.txt"

$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"

$ie = New-Object -com InternetExplorer.Application

$ie.visible = $true
$ie.silent = $false

$ie.navigate("$url")

while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}

$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()

start-sleep 5

$ie.navigate($url2)

start-sleep 10

$ie.Document.body.outerHTML | Out-File -FilePath $path$filename


This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.



After you have ran the script and are happy with the outcome you can then do the following.



Change the following so that IE does not open up on the desktop:



$ie.visible = $true


to



$ie.visible = $false


and add the following at the end of the script to ensure IE closes at the end:



$ie.quit()


Hope this helps.






share|improve this answer


























  • ... and how do you run it? ;-)

    – Hannu
    Jan 28 at 19:17











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1399245%2flogin-to-website-with-curl-and-download-webpage%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














As I was working on something similar at the time I quickly put this together for you:



Please change the path and filename to your requirements, make sure that the path has a trailing slash.



$path = "c:test"
$filename = "website.txt"

$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"

$ie = New-Object -com InternetExplorer.Application

$ie.visible = $true
$ie.silent = $false

$ie.navigate("$url")

while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}

$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()

start-sleep 5

$ie.navigate($url2)

start-sleep 10

$ie.Document.body.outerHTML | Out-File -FilePath $path$filename


This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.



After you have ran the script and are happy with the outcome you can then do the following.



Change the following so that IE does not open up on the desktop:



$ie.visible = $true


to



$ie.visible = $false


and add the following at the end of the script to ensure IE closes at the end:



$ie.quit()


Hope this helps.






share|improve this answer


























  • ... and how do you run it? ;-)

    – Hannu
    Jan 28 at 19:17
















1














As I was working on something similar at the time I quickly put this together for you:



Please change the path and filename to your requirements, make sure that the path has a trailing slash.



$path = "c:test"
$filename = "website.txt"

$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"

$ie = New-Object -com InternetExplorer.Application

$ie.visible = $true
$ie.silent = $false

$ie.navigate("$url")

while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}

$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()

start-sleep 5

$ie.navigate($url2)

start-sleep 10

$ie.Document.body.outerHTML | Out-File -FilePath $path$filename


This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.



After you have ran the script and are happy with the outcome you can then do the following.



Change the following so that IE does not open up on the desktop:



$ie.visible = $true


to



$ie.visible = $false


and add the following at the end of the script to ensure IE closes at the end:



$ie.quit()


Hope this helps.






share|improve this answer


























  • ... and how do you run it? ;-)

    – Hannu
    Jan 28 at 19:17














1












1








1







As I was working on something similar at the time I quickly put this together for you:



Please change the path and filename to your requirements, make sure that the path has a trailing slash.



$path = "c:test"
$filename = "website.txt"

$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"

$ie = New-Object -com InternetExplorer.Application

$ie.visible = $true
$ie.silent = $false

$ie.navigate("$url")

while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}

$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()

start-sleep 5

$ie.navigate($url2)

start-sleep 10

$ie.Document.body.outerHTML | Out-File -FilePath $path$filename


This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.



After you have ran the script and are happy with the outcome you can then do the following.



Change the following so that IE does not open up on the desktop:



$ie.visible = $true


to



$ie.visible = $false


and add the following at the end of the script to ensure IE closes at the end:



$ie.quit()


Hope this helps.






share|improve this answer















As I was working on something similar at the time I quickly put this together for you:



Please change the path and filename to your requirements, make sure that the path has a trailing slash.



$path = "c:test"
$filename = "website.txt"

$url = "https://url.retail.publishedprices.co.il/login"
$url2 = "https://url.retail.publishedprices.co.il/file/d/RamiLevi/"

$ie = New-Object -com InternetExplorer.Application

$ie.visible = $true
$ie.silent = $false

$ie.navigate("$url")

while($ie.ReadyState -ne 4) {start-sleep -m 100}
if ($ie.document.url -Match "invalidcert"){
$sslbypass=$ie.Document.getElementsByTagName("a") | where-object {$_.id -eq "overridelink"}
$sslbypass.click()
start-sleep -s 5
}

$ie.Document.IHTMLDocument3_getElementById("username").click()
$ie.Document.IHTMLDocument3_getElementById("username").value ='retalix'
$ie.Document.IHTMLDocument3_getElementById("password").click()
$ie.Document.IHTMLDocument3_getElementById("password").value ='12345'
$ie.Document.IHTMLDocument3_getElementById("submit").click()

start-sleep 5

$ie.navigate($url2)

start-sleep 10

$ie.Document.body.outerHTML | Out-File -FilePath $path$filename


This takes around 25 seconds to run due to the start-sleeps, this is to allow the pages to load. Depending on your connection you may need to increase these.



After you have ran the script and are happy with the outcome you can then do the following.



Change the following so that IE does not open up on the desktop:



$ie.visible = $true


to



$ie.visible = $false


and add the following at the end of the script to ensure IE closes at the end:



$ie.quit()


Hope this helps.







share|improve this answer














share|improve this answer



share|improve this answer








edited Jan 28 at 18:35

























answered Jan 28 at 18:26









CraftyBCraftyB

1,12429




1,12429













  • ... and how do you run it? ;-)

    – Hannu
    Jan 28 at 19:17



















  • ... and how do you run it? ;-)

    – Hannu
    Jan 28 at 19:17

















... and how do you run it? ;-)

– Hannu
Jan 28 at 19:17





... and how do you run it? ;-)

– Hannu
Jan 28 at 19:17


















draft saved

draft discarded




















































Thanks for contributing an answer to Super User!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1399245%2flogin-to-website-with-curl-and-download-webpage%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Mouse cursor on multiple screens with different PPI

Agildo Ribeiro

Sometime when accessing a menu: “Ubuntu 16.04 has experienced an internal error”