Polipo

2010-06-16
, , , , , , , , , , , , , , ,

Software > Proxies >

(on Wikipedia)
https://www.irif.fr/~jch/software/polipo/

A small and fast caching web proxy (a web cache, an HTTP proxy, a proxy server).

These are notes from when I tinkered with Ad-blocking with Polipo and adzapper.

—

Table of Contents [hide]

Usage ∞

\sudo  \killall  polipo
\sudo  \polipo  -c /mnt/320-data/polipo.config &

As a local web server ∞

– It can be used as an external web server too!

http://localhost:8123/

Uses the variable localDocumentRoot, which defaults to /usr/share/polipo/www/

polipo.config ∞

I don’t know that I made many changes, but here’s my polipo.config for reference.

Spoiler 
logFile = ""
logSyslog = false


# Sample configuration file for Polipo. -*-sh-*-

# You should not need to edit this configuration file; all configuration
# variables have reasonable defaults.

# This file only contains some of the configuration variables; see the
# list given by ``polipo -v'' and the manual for more.


### Basic configuration
### *******************

# Uncomment one of these if you want to allow remote clients to
# connect:

# proxyAddress = "::0"        # both IPv4 and IPv6
# proxyAddress = "0.0.0.0"    # IPv4 only

# If you are enabling 'proxyAddress' above, then you want to enable the
# 'allowedClients' variable to the address of your network, e.g.
# allowedClients = 127.0.0.1, 192.168.42.0/24
                                            
# allowedClients = 127.0.0.1              

# Uncomment this if you want your Polipo to identify itself by
# something else than the host name:

# proxyName = "polipo.example.org"

# Uncomment this if there's only one user using this instance of Polipo:

# cacheIsShared = false

# Uncomment this if you want to use a parent proxy:

# parentProxy = "squid.example.org:3128"

# Uncomment this if you want to use a parent SOCKS proxy:

# socksParentProxy = "localhost:9050"
# socksProxyType = socks5


### Memory
### ******

# Uncomment this if you want Polipo to use a ridiculously small amount
# of memory (a hundred C-64 worth or so):

# chunkHighMark = 819200
# objectHighMark = 128

# Uncomment this if you've got plenty of memory:

# chunkHighMark = 50331648
# objectHighMark = 16384


### On-disk data
### ************

# Uncomment this if you want to disable the on-disk cache:
diskCacheRoot = ""

# Uncomment this if you want to put the on-disk cache in a
# non-standard location:

# diskCacheRoot = "~/.polipo-cache/"

# Uncomment this if you want to disable the local web server:
localDocumentRoot = ""

# Uncomment this if you want to enable the pages under /polipo/index?
# and /polipo/servers?.  This is a serious privacy leak if your proxy
# is shared.

# disableIndexing = false
# disableServersList = false


### Domain Name System
### ******************

# Uncomment this if you want to contact IPv4 hosts only (and make DNS
# queries somewhat faster):

# dnsQueryIPv6 = no

# Uncomment this if you want Polipo to prefer IPv4 to IPv6 for
# double-stack hosts:

# dnsQueryIPv6 = reluctantly

# Uncomment this to disable Polipo's DNS resolver and use the system's
# default resolver instead.  If you do that, Polipo will freeze during
# every DNS query:

# dnsUseGethostbyname = yes


### HTTP
### ****

# Uncomment this if you want to enable detection of proxy loops.
# This will cause your hostname (or whatever you put into proxyName
# above) to be included in every request:

# disableVia=false

# Uncomment this if you want to slightly reduce the amount of
# information that you leak about yourself:

# censoredHeaders = from, accept-language
# censorReferer = maybe

# Uncomment this if you're paranoid.  This will break a lot of sites,
# though:

# censoredHeaders = set-cookie, cookie, cookie2, from, accept-language
# censorReferer = true

# Uncomment this if you want to use Poor Man's Multiplexing; increase
# the sizes if you're on a fast line.  They should each amount to a few
# seconds' worth of transfer; if pmmSize is small, you'll want
# pmmFirstSize to be larger.

# Note that PMM is somewhat unreliable.

# pmmFirstSize = 16384
# pmmSize = 8192

# Uncomment this if your user-agent does something reasonable with
# Warning headers (most don't):

# relaxTransparency = maybe

# Uncomment this if you never want to revalidate instances for which
# data is available (this is not a good idea):

# relaxTransparency = yes

# Uncomment this if you have no network:

# proxyOffline = yes

# Uncomment this if you want to avoid revalidating instances with a
# Vary header (this is not a good idea):

# mindlesslyCacheVary = true

Polipo ramdisk ∞

  1. I abandoned this, because I couldn’t get permissions to work for polipo.

  2. Instead, I disable polipo’s logging and disk cache.. done in my own polipo.config copied to /etc/polipo/config

  3. for polipo

  4. run as root

  5. /etc/fstab (fstab) would need:

  6. ramfs /tmp/ramdisk ramfs rw,size=100m 0 0

Spoiler 
_ramdisk=/tmp/ramdisk

\mkdir  --parents  $_ramdisk

# Seems to be shitty:
# \mkfs  -q  /dev/ram1  64000
# \mount  /dev/ram1  $_ramdisk

\sudo  \mount  -t ramfs  -o size=100m ramfs  $_ramdisk

# Fixme - reuse the cool code I use for _dotfiles
\rm  -rf \
  /var/cache/polipo \
  /var/log/polipo

\mkdir  --parents \
  $_ramdisk/polipo-cache \
  $_ramdisk/polipo-log

\touch  $_ramdisk/polipo-log/polipo.log

# Fucking hell..
\chown  -R  user:user  $_ramdisk
\chmod  777  $_ramdisk

\ln  --symbolic  $_ramdisk/polipo-cache  /var/cache/polipo
\ln  --symbolic  $_ramdisk/polipo-log    /var/log/polipo

2009-08-12 – 1.0.4 ∞

Firefox‘s skipscreen extension has issues on rapidshare.com. I regularly get an error message. It’ll happen on different files or the same files.. it’s on-and-off.

504 Connect to <url>:80 failed: Connection reset by client

The following error occurred while trying to access <url>:

504 Connect to url.rapidshare.com:80 failed: Connection reset by client
Generated Wed, 12 Aug 2009 01:14:25 PDT by Polipo on polipo:8123.
x

SkipScreen will download your file as soon as it is available!

~2009-08-11 – 1.0.4 ∞

Tested earlier than 2009-08-12.

Works just fine.

2009-07-16 – 1.0.4 ∞

While it seems to run, I’m having issues getting Firefox to actually use it properly. It needs to be troubleshooted further.

2009-05-08 – (1.0.4?) ∞

Installation ∞

  1. Download
  2. Uncompress
  3. make

  4. ./polipo

Configure your browser:

Firefox 3.0.8:

  1. Top menu: Edit > Preferences
  2. Advanced tab > Network sub-tab
  3. Settings button

  4. Manual proxy configuration

    1. HTTP Proxy: localhost
    2. Port: 8123
    3. Change “No proxy for” from localhost, 127.0.0.1 to be blank. I prefer this for a local server. You do as you wish.
  5. Save

  6. Visit _about:config_

network.http.pipelining                             true
network.http.proxy.pipelining                       true
network.http.pipelining.maxrequests                 15
network.http.max-persistent-connections-per-proxy   16
browser.cache.disk.enable                           false

Check these:

network.http.proxy.version                          1.1
network.http.proxy.keep-alive                       true
network.http.use-cache                              false

Update: I have some websites which do not work in polipo, and so I do keep cache enabled in Firefox for them.

Usage ∞

I start it with:

mkdir -p "/tmp/polipo/"

# I'm noticing too many broken websites..
#  pmmSize=8192 \
polipo daemonise=true \
  censorReferer=maybe \
  censoredHeaders="From, Accept-Language" \
  pidFile="/tmp/polipo.pid" \
  diskCacheRoot="/tmp/polipo/" \
  chunkCriticalMark=0 \
  chunkLowMark=0 \
  disableConfiguration=true \
  disableIndexing=false \
  disableLocalInterface=true \
  cacheIsShared=false 

# Purge the disk cache:
kill -USR1 `cat "/tmp/polipo.pid"`
sleep 1
polipo -x diskCacheRoot="/tmp/polipo/"
kill -USR2 `cat "/tmp/polipo.pid"`

I added this in my /etc/rc.d/rc.local (rc.local)

Information:

Stop it with:

kill `cat "/tmp/polipo.pid"`

Reload the forbidden URLs file:

  • write out all the in-memory data to disk (but won’t discard them)
  • reopen the log file

  • reload the forbidden URLs file

kill -s SIGUSR1 ``cat "/tmp/polipo.pid"``

Restart:

  • write out all the in-memory data to disk
  • discard as much of the memory cache as possible
  • reopen the log file

  • reload the forbidden URLs file

kill -s SIGUSR2 ``cat "/tmp/polipo.pid"``

Purge the disk cache:

kill -USR1 ``cat "/tmp/polipo.pid"``
sleep 1
polipo -x diskCacheRoot="/tmp/polipo/"
kill -USR2 ``cat "/tmp/polipo.pid"``

I added this in my /etc/rc.d/rc.local (rc.local)

Logging specified by logFacility, and the logging location is set with logFile, defaulting to /var/log/polipo

todo ∞

forbiddenFile (defaults to ~/.polipo-forbidden or /etc/polipo/forbidden, whichever exists) specifies the set of URLs that should never be fetched. If forbiddenFile is a directory, it will be recursively searched for files with forbidden URLs.

Every line in a file listing forbidden URLs can either be:

  1. a domain name
  2. a string that doesn’t contain any of /, * or \ —

  3. or a POSIX extended regular expression.

Blank lines are ignored, as are those that start with a hash sign #.

  • easylist won’t work as a forbiddenFile.

    • Convert the first character of ! to #
    • Anything without a / * or \ should be ok as-is, but Convert everything else to a proper regular expression.
  • Get Firefox’s type-and-search feature going again.. it won’t work on single words. =/

  • Somewhere in the Polipo website there is a mention of a smarter cache cleaning tool. Get/use it. Otherwise rig a cron job to purge things.

notes ∞

A number of websites incorrectly mark variable resources as cachable; such issues can be worked around in polipo by manually marking given categories of objects as uncachable. If dontCacheCookies is true, all pages carrying HTTP cookies will be treated as uncachable. If dontCacheRedirects is true, all redirects (301 and 302) will be treated as uncachable. Finally, if everything else fails, a list of uncachable URLs can be given in the file specified by uncachableFile, which has the same format as the forbiddenFile (see Internal forbidden list). If not specified, its location defaults to ‘~/.polipo-uncachable’ or ‘/etc/polipo/uncachable’, whichever exists.

sites which won’t work with it ∞

  • nonoba.com

    • inconsistent issues with the style showing up and with the flash portion of a game page not appearing.

  • groups.google.com

    • inconsistent issues with refreshing into a blank page, and with the style not showing up on cached versions.

  • www.linuxquestions.org

    • inconsistent failure to load pages. I got an infinite redirect loop.
1 comment

One comment on “Polipo”

Leave a Reply

Your email address will not be published. Required fields are marked *