Decrypting SSL ‘protected’ web traffic for further analysis (PoC)

Print Friendly

These are the brief steps I did to setup a SSL MiTM(man-in-the-middle) PoC(proof-of-concept) that is pseudo-in-line, in case I need to refer to it, on my own blog.

I intend to leverage this PoC to integrate some logging facility e.g. SIEM, database for mining, malware behavior analysis and etc. Other interested projects which I read briefly are OpenFWTK and OSSIM. Apologies if the above steps are not complete for those interested to try. I just followed documentations from the links provided in the References below, hoped they can be helpful, especially the mailing lists.

The PoC is simply illustrated as:
[1]client PC --> [2]Proxy (Squid)==[3]AV(C-ICAP) --> [4]Websites(HTTPS/HTTP).

[1] Client initiates connections to websites e.g. https://gmail.com, https://facebook.com, http://google.com.
[2] Proxy communicates with websites on behalf of Client. Decrypt(MiTM) if SSL traffic, else just act as any other HTTP proxy service.
[3] Proxy also communicates with AV server via ICAP protocol. AV server will analyse the requests from Client and/or responses from Websites including files.
[4] Websites are public Web 1.0 and 2.0 services.

An objective is to show SSL termination using FOSS (Free/Open Source Software) Squid so that HTTPS websites are decrypted for further analysis e.g. virus.
This article does not go into other related concepts e.g. transparent proxy, dnsmasq, browser exceptions handling etc.

A) Ready a plain Linux distro. I use CENTOS 5.X base installation with development tools and libraries.

B) Download latest clamav (clamav-0.96.tar.gz), c-icap (c_icap-060708rc3.tar.gz) and squid (squid-3.1.3.tar.bz2) source codes. I use C-ICAP as a platform to jump start the ICAP service, any SQUID compatible ICAP service could be used.

C) cd /opt ; mkdir squid-build clamav c_icap.

D) SQUID compilation (to note are –enable-ssl and –enable-icap-client):
./configure --enable-icap-client --enable-follow-x-forwarded-for --enable-storeio=aufs
--prefix=/opt/squid-build/ --disable-ident-lookups --enable-async-io=100 --enable-useragent-log
--enable-kill-parent-hack --enable-forw-via-db --enable-ssl
make && make install

E) CLAMAV compilation:
./configure --prefix=/opt/clamav/ --enable-bigstack --enable-readdir_r--enable-clamdtop --enable-llvm
make && make install

F) C-ICAP compilation:
./configure --enable-large-files --prefix=/opt/c_icap/
make && make install

G) Update clamav to get the latest signatures. This is not vital to the PoC but leave a note here.
/opt/clamav/bin/freshclam

I) Setup Squid proxy – /opt/squid-build/etc/squid.conf (To note that squid.pem is created using openssl package installed in CENTOS)
http_port 3129 sslBump cert=/etc/pki/tls/certs/squid.pem
ssl_bump allow all
http_access allow all
always_direct allow all
sslproxy_cert_error allow all
icap_enable on
icap_preview_enable on
icap_preview_size 128
icap_send_client_ip on
icap_send_client_username on
icap_service service_req reqmod_precache bypass=1 icap://localhost:1344/srv_clamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache bypass=1 icap://localhost:1344/srv_clamav
adaptation_access service_resp allow all

J) Setup C-ICAP – /opt/c_icap/etc/c-icap.conf
Did not do anything much except modify the “User” and “Group” that daemon (process) will run with.

K) Start C-ICAP
/opt/c_icap/bin/c-icap

L) Start Squid
/opt/squid-build/sbin/squid

M) Open Firefox/Chrome (any browser), set proxy to IP address of the PoC and port “3129” for all protocols.

N) View “https://gmail.com”, “https://yahoo.com”, “http://google.com”. Accept any SSL/security exception for PoC.

O) View “http://www.eicar.org/anti_virus_test_file.htm” and test opening "http://www.eicar.org/download/eicar_com.zip" and "https://secure.eicar.org/eicar_com.zip".
This is to show that the AV server can analyse both HTTP and HTTPS (SSL 'protected') traffic. Using the default setup, I got the following message on browser:
"VIRUS FOUND

You try to upload/download a file that contain the virus
Eicar-Test-Signature
This message generated by C-ICAP/060708rc3 srvClamAV/antivirus module”,

"... ... general, VIRUS DETECTED: Eicar-Test-Signature." in the c-icap server.log,
and this "... ... TCP_MISS/403 991 GET http://www.eicar.org/download/eicar_com.zip - DIRECT/188.40.238.250 text/html" in the squid access.log.

If AV server is down (e.g. killall c-icap), the file will be downloaded until it C-ICAP is brought online again.

P) Using tcpdump, I could verify whether HTTPS pages are indeed decrypted and forwarded in clear to from Proxy to AV.
[root@localhost /tmp]# tcpdump -i lo -Xnvvv -s 0 dst port 1344 | grep pilot
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
0x03c0: 696c 6961 6e20 7069 6c6f 7420 7768 6f20 ilian.pilot.who.

*To note that I was browsing “https://xxx.gov” which has the words “civilian pilot who” in the SSL protected page.

References:
1. C-ICAP – http://sourceforge.net/projects/c-icap/
2. CLAMAV – http://www.clamav.net
3. SQUID – http://www.squid-cache.org

3 Responses to “Decrypting SSL ‘protected’ web traffic for further analysis (PoC)”

Read below or add a comment...

  1. mamak says:

    You forgot to create clamav user and group before compiling

  2. Fingers says:

    Thanks very much for this wonderful post.

Leave A Comment...

*