Custom Search
Number of hits:

Monday, January 21, 2008

Break / Bypass download limit !! Hack the squid proxy :-)

Many http proxies enforce a download limit to users.e.g no downloads over 5MB
It is really horrible... if you can download 10 files of 1MB this is simply $%%%

while trying to figure out a workaround I found this very simple method...
why not fool the proxy saying we are downloading only small files ?

Here's a small script to do it. It works with squid proxies.
Download it from here

I wrote its initial version in C. However, it became quite difficult to install because of externel dependancies. But this will work if you have perl & curl which all systems usually have...

If you like this pls link to this page and make this popular... I'd like to see people removing this stupid download limit thing and make the www free !

enjoy breaking limits !!

pls report dead link to indikabandara19@(yahoo|gmail).com

93 comments:

Ensilage said...

Dear Sir/Madam
I like your idea in this project. but do not know how to configure or run it or how does it works. Even i am not gud in perl but i need it because my user account in my college has been blocked because of downloading more than 5omb. Our college squid proxy download limit is 50mb and there is some setting like how much time spent these are also calculated but i am a regular internet user i always uses 4days in a weak in internet how can I get rid from it and tell me detail how to run or how to configure this. My account has already blocked i need it back can i get it.

Indika said...

if your school computer is a linux i'm sure it has perl and a command line tool curl is installed...
if you're on windows just install cygwin. you won't have any problem installing cygwin 'cos ur limit is 50MB ! can u imagine mine is only 5!!
in cygwin select perl and curl and other dependencies the setup complains about..
fire up a command line.. download the script there. the README is on top of the script !!!

Bharat kumar said...

Hi the script u wrote is great.can u plz tell me how to use this for an authenticated proxy which requires user name and password.If i want to use this in windows is Cygwin compulsory or is there any other alternative bcoz i dont want to install Cygwin in my system.can u suggest some other alternative to cygwin and also how to use with authenticated proxy.

Bharat kumar said...

if i install perl and curl in my windows machine can i use ur script instead of using cygwin.

Indika said...

I'm not sure whether u can install curl on windows. There is perl however.

there's a configuration section in the script modify the variable $_proxy_auth as to adjust your user:pass

kanoe said...

hi ,im from indonesia
sorry about my bad english.
i have my local squid on my pc ,that chace_peer ing to my isp squid where limited my download only 15MB .
how i must implement your perl script to my local squid so my local squid can tell to isp squid that my squid request is not more then 15MB .
tanks before

Indika said...

I guess ur setup is
isp-squid <-> ur-squid <-> you
ur-squid is no different than isp-squid ie. the requests you send to ur-squid will be sent to isp-squid as it is. So what you can do is use the script to connect to ur-squid ie. set proxy:port to ur-squid and download. Did it not work for u?

Amal said...

Hi.. I'm on Vista,
My university connection does not even allow me to download stuff like picasa. !!
Also I cannot download Cygwin.

Do you have any version of this script that runs directly from Windows?

Thanks a lot.

Indika said...

is it you have a download size limit or other limitation eg. bandwidth ...

well, i downloaded cygwin with this script!!
install cygwin with minimum stuff(u won't require large packages eg. >5MB)
make sure u install perl and curl

or

try to install a perl and curl windows version

Amey said...

Hi,
Despite initializing the script with a zip file, it continued to insist that I initiaze it whenever I tried to download a zip file, and would refuse to go any further. So I removed the check, and it worked fine :)
It's downloading the file right now, so don't know whether it'll be successful... The script seems to download the file in roughly 1 mb chunks..
Brilliant piece of work!!

Amey said...

It worked!!
You, madam, are a genius!!

Indika said...

btw, i'm not a madam :D ... indika is a male name in SL

Amey said...

oh crap, so sorry :P

Amey said...

I was wondering.. would it be possible to run parallel downloads using your script? I mean, split the file into parts the way it already does, and download, say, three parts in parallel instead of sequentially.. I don't know much about perl, otherwise I would attempt it myself..

Indika said...

yes indeed..eg. create a few threads and ask them to download the parts and later concatenate together..

I was hoping to do that but unfortunately don't have much time :-)

I'm definitely sure that you can do this elegantly and easily with python

ady said...

Its not working for me...Can sum1 plz help?

ady@MIT04-54 /cygdrive/c/script
$ perl pget.pl -i C:\Users\ady\Desktop\ByPassProxy\setup.exe
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.

binmode() on closed filehandle $f at pget.pl line 246.
sysread() on closed filehandle $f at pget.pl line 247.
I handle
exe -> 0

ady@MIT04-54 /cygdrive/c/script
$ perl pget.pl -g http://www.cygwin.com/setup.exe
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.

Unknown ext in setup.exe. Rerun with -i fileName at pget.pl line 138.

ady@MIT04-54 /cygdrive/c/script
$

Indika said...

it seems pget was not initialized properly and/or the path for pget.ext is not what is mentioned in the script.
make sure the $_extFile config in the script is correct. open that file and check whether there are entries like below

gz 31
exe 77

ady said...

The first time i ran the script with -i option is gave me the msg: extFile is not at %s. Creating one
(Dont remember xactly what stood for%s)

My understanding of UNIX/Linux and perl is very limited.

Can you plz explain to me what the -i option does. And the file name that has to be given to the -i option as a parameter. What type of file it has to be?
My understanding is that If I have to download a mp3 file i wud give an mp3 file on my Hard drive with completed path as a parameter to -i option. Is it correct?

What is pget.ext file for?
the pget.ext file i have contains just 1 entry:
exe 0

Thanx for help
-Aditya

Indika said...

>>
My understanding is that If I have to download a mp3 file i wud give an mp3 file on my Hard drive with completed path as a parameter to -i option. Is it correct?
>>
yes u r correct


>>
the pget.ext file i have contains just 1 entry:
exe 0
>>
this entry is incorrect.
change to
exe 77

this should be automatically done with -i option for given file type (exe, mp3 etc) provided that u have supplied a valid path for $_extFile config

ady said...

okkk.. i got it working... had to tweak a few http headers...
Now i am able to download files greater than 5MB... but theres still 1 prob...
The files i download thru this perl script in the cygwin env.... if i try opening them thru a windows application, i get an error that the file is corrupt .... any idea whats going wrong??

ady said...

it was a file permission problem.. resolved!! :)

btw how do u manage to capture urls of the stuff u wanna download... ??
many few thing now are available as static urls..

daibatzu said...

Hi, I'm getting the same error ady was getting where it says:

Unknown ext in setup.exe. Rerun with -i fileName at C:\prog\pget.pl

I'm running this on windows. Any idea what to do? He said he modified http headers and I have no idea what headers to modify. pget.ext is in the right directory (at least to my knowledge)
Thanks

daibatzu said...

oops, finally got it working only to find out that my squid proxy is 2.4

It says:

Generated Sun, 05 Oct 2008 22:25:38 GMT by localhost (Squid/2.4.STABLE7)

Aarghhhhh

Indika said...

if squid 2.4 can impose a limit it should work too... i dont think they changed the way how this limit is applied from 2.4 to 2.6

daibatzu said...

Maybe my limit is not even up to 1mb exactly. If I wanted 500kb chunks, what do you recommend I set my chunk size to?

daibatzu said...

Ok this is what I've done. I changed the variable my $_chunkSize = 750*1024;

So that it downloads 750kb chunks. The problem is that it has downloaded only 750kb of the file. Is there something else I need to change in pget.pl so that it downloads the remaining chunks as well?

Indika said...

may be the proxy doesn't handle partial get ?
can u try with changing the chunk size ?

Amey said...

Hey, I'm back :P
I'm having problems.. Like I mentioned, I was having problems with the script asking for initialization of extensions even though I'd initialized it, so I removed that check. But I still initialize it before I download anything. Now, pget.ext contains exe 77, but when I download an exe, the file downloaded doesn't work. It has a generic windows executable icon, and double-clicking on it doesn't do anything.
Any idea what's wrong?

Indika said...

if it still complains not initialized (u disabled that?) it may not be picking up the correct configuration file. so, you may have to point to the correct file by changing the _extFile option. (u can remove the warnings but if there is an error it will still be there ! )

Amey said...

See, I redownloaded your script, modified the proxy settings and left everything else unchanged. Here's the console output I get:
----------
$ ./pget.pl -i part1.exe
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.
I handle
exe -> 90
----------
And then...
----------
$ ./pget.pl -g http://ftp.riken.jp/Linux/suse/projects/go-oo/stable/win32/3.0/GoOo-3.0-19.exe
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.
Unknown ext in GoOo-3.0-19.exe. Rerun with -i fileName at ./pget.pl line 138.
-----------
I've removed the tags around "fileName" since Blogger wasn't letting me keep them..
So, what's going wrong? If it adds the extension to pget.ext, it's obviously not having problems finding the file, so what's wrong?

Indika said...

change the config for _extFile to use a full path not a relative path
ie.
change
my $_extFile = "./pget.ext" ;
to
my $_extFile = "/home/amey/pget.ext" ;

btw, the entry in pget.ext is wrong. it is usually 77 for exe

Amey said...

Hi,
Did that, it changed the entry to 77, but I'm still getting the same message when i try to download any exe..

Indika said...

I uploaded the script i'm currently using. It works for me in the same scenario as you (perl cygwin).

If this too fails pls send me a mail with details of your setup/script with ur config/ext file etc.

Amey said...

Hi,
The proxy at my college is unfortunately a censorship proxy as well, so I'm unable to access your blog post directly. I access it via google's cached version of the page, and it'll take a few days for google to upload it's cache, so could you please mail me the script you just uploaded? My address in email.amey[at]gmail.com

Jck said...

my campus has capped the download limit ( i can download at a maximum speed of 5 kbps). is there anyway i can bypass this?

Kal said...

Hi Indika,

Thanks for ur excellent piece of work. It's amazing how powerful these unix utils are. I successfully downloaded a b2z file and I try to download iso image too. But ext for iso is getting the value 0. Where is these values comes from? Is there any kind of list of file types that we can put for ext file rather than feeding it an actual file? what is this process..? I'm sorry I don't have that much of perl knowledge, but I start learning it :D

Thanks alot again..

Kal

Indika said...

Good to hear it worked :)
it is 0 for iso and you don't have to feed an actual file each time. once is enough for one file type.
here is my ext file


tar 67
bz2 66
wmv 48
rpm 237
WMV 48
rar 82
iso 0
msi 208
gz 31
mp3 73
pdf 37
tgz 31
flv 70
zip 80
exe 77

muaz said...

Thank you for this wonderful script. However, I still cant bypass the proxy. I know the limit is 10mb, so I set to this:-

my $_chunkSize = 750*1024;

Whenever I tried to download larger file, I'll get this error:-

Based on your corporate access policies, access to this web site or download ( http://rlink.hopto.org/files/%5BRLINK%5D_________.zip ) has been blocked because the download size exceeds the allowed limit.

If you have questions, please contact your corporate network administrator and provide the codes shown below.
Notification codes: (1, FILE_SIZE, BLOCK-SIZE, http://office.org/files/%5BRLINK%5D_________.zip)

Any way to bypass this?

Indika said...

1. What's ur proxy eg. Squid/Microsoft?

2. Does this error occur for any file?

muaz said...

I'm not sure which proxy. May I know how to check for what type of proxy my company are using?

And it happened to all file above 10mb. Below that, it works fine

Thanks

289501 said...

I m not a programmer. So i can't understand perl or cygwin.
I have windows vistax64(ultimate). I installed cygwin full as u said and then installed perl. I copied ur script to nopepad and saved it as pget.pl.
Then I double clicked on then pget.pl file and nothing happened. I checked the downloading status and still i m unable to download file larger than 80 mb.
Plz help me.
And plz plzz explain me the complete procedure.
My college has squid proxy 2.6 with auth pass word and max download limit of 85 MB.

Indika said...

how to run
1. save pget.pl script to your home
2. open the file with a text editor and read the README and USAGE sections
3. pget.pl is runnable after you enter
chmod +x pget.pl
on the command prompt
4. you run it with ./pget.pl

289501 said...

as u said i entered the command "chmod +x pget.pl" in command prompt but it didnt worked. Also i am unable to install curl on my windows. What is the role of cygwin in running the pget.pl??
I got installed perl on my windows.
Sry for my very little knowledege about programming.

289501 said...

And also i want to ask what is php??
i read that to installed curl on windows u need to install pp and apache first!!

Amey said...

Hi,
You wouldn't happen to have another script for getting irc, git and ssh connections throught squid, would you? :/

Industriers said...

hello i'm krishna from indonesia,

i'm really grateful found your tutorial, i really need this. Since i'm not a progammer and even dont know network i still dont understand how to use your script.
Here they are the questions :
well, mine is running on vista business sp1, wifi network, when get only need through VPN account, and then setting the proxy at browser. All i know the proxy server at my campus is squid.
And know i still dont understand the steps that you posted.
1. should i install the perl n curl, how to check them whether they have been installed?
2. the bandwidth limit at my campus perhaps 18MB per file, should i change the chunks? i dont understand what chunks is...
3. the other configuration is proxy, i know my proxy, but i dont think they need username and password, so shall i let them blank? it's different from VPN account right?
4. i still dont understand how to configurate
my $_extFile = "./pget.ext"
and
my $_curl = "/usr/bin/curl";
what does their function and what for?

5. can i download the file with internet download manager? because with that application, the speed is become awesome...

6. can i do multiple download at the same time? with download manager?

thank you bro, i'm really apreciate it, sorry for my bad english...

Sir LMP said...

sira....ayye keep it up
eth windows xp wala meka karanna widiyak thiyennawada?

Indika said...

cygwin

Sujith said...

Ela kiri indika ayye...



Sujith

saddan said...

Hey, great script, I've made a modification since I noticed some servers consider 0 as the first byte.
I'll send to your email.

munish said...

Sir, I am on a network which uses a proxy server to connect to the internet. My computer is a client computer on the network. The server proxy allows us to connect to the internet only during certain period. However if i start a download during this period, it doesn't stop even after the time is up on proxy server. No browser or application is able to connect now to the internet. However the download started during the allowed period keeps going on until complete. Is there anyway to stay connected to the internet even after the time is up on proxy server.

abish said...

abish@abish-lappy:~$ perl pget.pl -i http://download.softpedia.ro/dl/393560c4e7f6823a7795b006596de2fc/4aad9c20/100006527/software/ANTIVIRUS/avira_antivir_personal_en.exe
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.

known ext
Couldn't open http://download.softpedia.ro/dl/393560c4e7f6823a7795b006596de2fc/4aad9c20/100006527/software/ANTIVIRUS/avira_antivir_personal_en.exe at pget.pl line 290.


what is wrong?
using ubuntu 9.04 behind squid.

Indika said...

Hi abish,
To download you should run with '-g'
option. Pls read the README section on how to use the script

abish said...

abish@abish-lappy:~$ perl pget.pl -g http://download.softpedia.ro/dl/393560c4e7f6823a7795b006596de2fc/4aad9c20/100006527/software/ANTIVIRUS/avira_antivir_personal_en.exe
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.

known ext
Unknown ext in avira_antivir_personal_en.exe. Rerun with -i fileName at pget.pl line 153.
abish@abish-lappy:~$

:) sorry! didn't work. Thank you for taking the trouble.

abish said...

abish@abish-lappy:~$ perl pget.pl -g http://mirror.lupaworld.com/ubuntu/releases/jaunty/ubuntu-9.04-desktop-i386.iso
pget version 0.1.0
There is absolutely NO WARRANTY for pget.
Use at your own risk. You have been warned.

known ext
iso, 0
exe, 0
Use of uninitialized value $_proxy_auth in concatenation (.) or string at pget.pl line 321.
/usr/bin/curl -x 10.0.0.1:6588 -u -r 1-1048576 -# "http://mirror.lupaworld.com/ubuntu/releases/jaunty/ubuntu-9.04-desktop-i386.iso"
Enter host password for user '-r':
######################################################################## 100.0%
######################################################################## 100.0%
Downloaded ubuntu-9.04-desktop-i386.iso(2117 bytes).
abish@abish-lappy:~$
? the file is not 2117 bytes only.. where did i go wrong?
please help,

Indika said...

hi abish:
did u remove the proxy_auth values?
even if you dont require proxy authentication leave them as they are.

abish said...

thanks :)
i am a moron.
you've created a smart script.
wondering if we could use this in rapidshare and 4shared.

anyways, thanks a lot!

MICKY said...

hii myself virinder
i'm in coll they r using squid proxy with uthentication giving us net only few hours and download limit 20 mb plz help me to solve my prob

ravi said...

hello there iam a new user to your pget.pl i just want to download avi movie file but it does not download like when i downloaded from http://web.mit.edu/ccobrien/Public/Moulin%20Rouge.avi i did not get my full movie but around 2 to 3 kb
can you tell me what is the problem

ravi said...

my chunsize is 1024* 1024
so should i change it to something else

thanks in advance.

ravi said...

pls reply as soon as possible

Indika said...

Pls email me of ur problem

munish said...

is there anyway to accelerate the download speed.. i mean more number of parallel connections?

satish said...

hello sir,,,, i didnt understand how to configue,,, please can u give me tutorial on how to use this code....i m noob i dont know all these if u can give me a detailed procedure i m very helpfull to yuu

satish said...

hello sir,,,, i didnt understand how to configue,,, please can u give me tutorial on how to use this code....i m noob i dont know all these if u can give me a detailed procedure i m very helpfull to yuu

Arnav Arora said...

hi ur script is great..
i have it all configured but when i download a 200 Mb file with 2Mb chunks after some downloading i get curl: (18) transfer closed with some outstanding read data..
Plz tell how to fix this

Anonymous said...

link's not working
pls help
thanks

Anonymous said...

sir
i followed the instructions but i cant download more then 1 kb even when im downloading file of greater size.
i have left proxy_auth as it is because we dont reqiure it here.

PS: cant we download rar file through ur scipt
thanks in advance

aNkit said...

I have done everything as said. But am getting this error:

./pget.pl -i trial.zip
bash: ./pget.pl: /usr/bin/perl^M: bad interpreter: Permission denied

Anonymous said...

thanks dude.. your script works smart on me.. :)

Kian said...

Hey! Thanks for the script.
I can download nicely now.
but I cant download from hp ftp server..
It just downloaded one chuck and it stop. I think its because it cant resume.
Is there a way to fix this?

Kian said...
This comment has been removed by the author.
Indika said...

afaik, ftp doesn't provide partial download. so the script won't work with ftp servers ....

Kian said...

owh..
I was able to download a smaller file off the ftp server using a download manager. It managed to run 2 streams together, so i assumed it does support resume.

I opened the downloaded file. Its actually an html. It reported that my login was incorrect.

Also

Your script added user@ in front of the url. Just pointing it out, in case that isnt normal.

Kian said...
This comment has been removed by the author.
Kian said...

i tried changing the user:pass to the one used in my download manager.
anonymous:IEUser@

I got a request is too large error
=(

Kian said...

Haha! i managed to download the driver after all!
just change the ftp:// to http://
Probably doesnt work with all ftps.
The download was unstable with lots of pauses. But it came out uncorrupted!!
Nice job on the script

Anonymous said...

can u explain how this work?

sami said...

hi indika,
your script is great, i have gone through the whole process but there still some problems,
1.my univercity is using (squid/2.7.STABLE7),
2. i get a value in exe 73, how can i change it to 77
3. my domnloaded file is only 4kb,

is there any solution plz??
thanks

Indika said...

@sami,
1. squid version wouldn't matter (i hope)
2. edit the pget.ext file by hand (not recommended) or use a different exe file and rerun initialization
3. send me the downloaded file

sami said...

my univercity squid proxy do not permit ping ,telnet etc services, moreover i can't ping from one department to another department..any bypass solution plz??

Anonymous said...

Dont cheat us

Anonymous said...

i got new solution.
internet download manager is compatible with cygwin.
i dont need to make the extension file.
idm simply grab the file.
the main point of cygwin is to fool the squid saying its 1 mb right.

Arin said...

how do i use cookie with this kind of script?
will it work if the download link requires username & password combo?
i mean it dwnl link is "http://username:password@****.com"

riz3via said...

i got ERROR: curl is not at usr/bin/curl. Please install or get correct path

how do i fix this?

guru said...

I am a client with a squid proxy server in my college. All normal PCs are limited to download speed of about 20kbs. Is there anything one can do to break this. Thanx for ur previous script

rucciva said...

Hi indika,, what is the right index (i mean the one in the pget.ext file) for mp4 file,,,

coz when i run it with -i options, it produced 0 for mp4 file and i can't play my file,,,

thanks

Anonymous said...

need a setup..plz attach setup of your software..?

Anonymous said...

[url=http://sexrolikov.net.ua/tags/%EC%EE%EB%EE%E4%E5%ED%FC%EA%F3%FE/]молоденькую[/url] Online porno : [url=http://sexrolikov.net.ua/tags/%EF%E0%F5%F3%F7%F3%FE/]пахучую[/url] , все это смотреть

Anonymous said...

top [url=http://www.001casino.com/]free casino[/url] hinder the latest [url=http://www.realcazinoz.com/]free casino[/url] manumitted no store perk at the foremost [url=http://www.baywatchcasino.com/]redeem casino
[/url].

Anonymous said...

Fabulouѕ, what a weblog іt is! Thіs ωebѕite provіdеѕ useful data tо us, keep it up.



Αlso visit my weblοg - wso launch

Anonymous said...

Thank you fοr the auspіcious wгitеuр.
It in fact wаѕ a amusement account it.
Look аԁvanced to far аdded аgreеablе fгоm you!
By the ωay, how can we communicаte?

my web blog ... get cash for surveys review scam

Anonymous said...

carefully It should be kept away from the interest rate This rate is established by [url=http://www.kukuhot.com]ブランド財布[/url] interest grows in something outside of your its laws change constantly The key to [url=http://www.kukuhot.com]ブランドコピー[/url] enough the gallery has 40 full color pages this visa are that aside from allowing the [url=http://www.kukuhot.com]バッグコピー[/url] the one with my partner 16When asked what peine de passer plus de 10 minutes devant http://www.kukuhot.com[/url] points or virtual currency that can never Charge Former Satellite Radio Worker