Search Support

Avoid support scams. We will never ask you to call or text a phone number or share personal information. Please report suspicious activity using the “Report Abuse” option.

Learn More

How do you log/save the actual pages which are part of a session?

  • 12 replies
  • 2 have this problem
  • 1 view
  • Last reply by cor-el

more options

based on the information given:

https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging
you may thoroughly log the inner workings of HTTP sessions as handled by the browser.
I also need to save and keep the actual pages that are being downloaded by the browser (including the supporting css, javascript, images, ...)
How do you achieve that?
lbrtchx
based on the information given: https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging you may thoroughly log the inner workings of HTTP sessions as handled by the browser. I also need to save and keep the actual pages that are being downloaded by the browser (including the supporting css, javascript, images, ...) How do you achieve that? lbrtchx

Chosen solution

OK, I was finally able to do what I wanted.

Here is an outline for anyone to benefit from, improve it:

// __ latest firefox development version (aurora)

https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/firefox-52.0a2.en-US.linux-i686.tar.bz2

$ ls -l firefox-52.0a2.en-US.linux-i686.tar.bz2 -rw-r--r-- 1 knoppix knoppix 65124511 Dec 27 05:31 firefox-52.0a2.en-US.linux-i686.tar.bz2

$ md5sum -b firefox-52.0a2.en-US.linux-i686.tar.bz2 0877480001d91ebdb180011db8f84df5 *firefox-52.0a2.en-US.linux-i686.tar.bz2

// __ untar/install it

$ _LOG="tar_-xjvf_firefox-52.0a2.log"

$ date; time tar -xjvf firefox-52.0a2.en-US.linux-i686.tar.bz2 > "${_LOG}" Wed Dec 28 04:09:02 EST 2016

$ wc -l "${_LOG}" 78 tar_-xjvf_firefox-52.0a2.log

$ ls -l "${_LOG}"

// __ run/test it

$ cd firefox

$ pwd .../firefox

$ ./firefox --version Mozilla Firefox 52.0a2

// __ While developing, I find useful running tcpdump, wireshark and firefox in logging mode

// __ run firefox and set up logging via about:config

_DT=$(date +%Y%m%d_%H%M%S)

_LOGFL="${_ODIR}/${_DT}_firefox-52.0a2_MOZ.log" echo "// __ \$_LOGFL: |$_LOGFL|"

export MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5

export MOZ_LOG_FILE="${_LOGFL}"

./firefox &

Read this answer in context 👍 0

All Replies (12)

more options

You can try Wireshark.

more options

Try wireshark for that? Firefox is already nicely logging the HTTP sessions

What I need is for firefox not to delete the data feeds of html pages, js, ... as it finishes to process, show them.

There should be some sort of "about:cache_duration" or "about:cache_length" option,

What is it?

Modified by Albretch

more options

Probably, another MOZ_LOG parameter would do, but I don't know what would it be.

MOZ_LOG parameters don't look like about:config parameters

to me.

more options

Modified by cor-el

more options
See:
yes, but in that document it says:
Using about:networking This is available starting with Firefox 52. Navigate to about:networking and go to the Logging section. ...
and the most up to date version of firefox is 50.1 right now 

and the beta release is 51.0beta:

https://www.mozilla.org/en-US/firefox/50.1.0/releasenotes/ https://www.mozilla.org/en-US/firefox/51.0beta/releasenotes/
so I am not getting something right or you guys are 

not being clear enough to me.

There must be a way to do such a thing, which you 

are using while debugging  ;-)

Modified by Albretch

more options

You would have to install the Firefox Developer version (52.0a2) to set all parameters from within Firefox via about:networking. In the current release you only can do that via environment variables.


#!/bin/sh

# https://developer.mozilla.org/en/HTTP_Logging
# *https://developer.mozilla.org/en/Mozilla/Developer_guide/Gecko_Logging

# try one of these (remove the #)
# export MOZ_LOG=nsHttp:3
# export MOZ_LOG=nsHttp:5

export NSPR_LOG_FILE=~/loghttp.txt

# set the correct path to Firefox
cd /usr/local/firefox-RE

# start Firefox
./firefox -no-remote -P
more options

Albretch said

I also need to save and keep the actual pages that are being downloaded by the browser (including the supporting css, javascript, images, ...)

I doubt that logging will give you a convenient format. I once turned on logging for something. After 25 seconds, I had a text file of nearly 392,000 lines consuming over 42 megabytes. The responses were encoded (sent with gzip encoding) so they are indecipherable -- if they are included in the file, which I can't confirm. Every so often I find decoded response headers but I don't see the actual response content (text or image, as the case may be).

Hopefully there is another way, such as an add-on, that is simpler than fishing around in the cache.

more options
more options

Chosen Solution

OK, I was finally able to do what I wanted.

Here is an outline for anyone to benefit from, improve it:

// __ latest firefox development version (aurora)

https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/firefox-52.0a2.en-US.linux-i686.tar.bz2

$ ls -l firefox-52.0a2.en-US.linux-i686.tar.bz2 -rw-r--r-- 1 knoppix knoppix 65124511 Dec 27 05:31 firefox-52.0a2.en-US.linux-i686.tar.bz2

$ md5sum -b firefox-52.0a2.en-US.linux-i686.tar.bz2 0877480001d91ebdb180011db8f84df5 *firefox-52.0a2.en-US.linux-i686.tar.bz2

// __ untar/install it

$ _LOG="tar_-xjvf_firefox-52.0a2.log"

$ date; time tar -xjvf firefox-52.0a2.en-US.linux-i686.tar.bz2 > "${_LOG}" Wed Dec 28 04:09:02 EST 2016

$ wc -l "${_LOG}" 78 tar_-xjvf_firefox-52.0a2.log

$ ls -l "${_LOG}"

// __ run/test it

$ cd firefox

$ pwd .../firefox

$ ./firefox --version Mozilla Firefox 52.0a2

// __ While developing, I find useful running tcpdump, wireshark and firefox in logging mode

// __ run firefox and set up logging via about:config

_DT=$(date +%Y%m%d_%H%M%S)

_LOGFL="${_ODIR}/${_DT}_firefox-52.0a2_MOZ.log" echo "// __ \$_LOGFL: |$_LOGFL|"

export MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5

export MOZ_LOG_FILE="${_LOGFL}"

./firefox &

Modified by Albretch

more options

then the wireshark and tcpdump part

// __ You don't want to run wireshark as root!

$ sudo dpkg-reconfigure wireshark-common

$ sudo adduser -a $USER wireshark

// __ Set up logging file names for easy management

_NM="..."

_DATA_DIR=".../data"

_ODIR="${_DATA_DIR}/${_NM}"

mkdir "${_ODIR}"

_DT=$(date +%Y%m%d_%H%M%S)

_WSLOG="${_ODIR}/wireshark-gtk_${_DT}.log" echo "// __ \$_WSLOG: |$_WSLOG|"

// __ running wireshark

sudo wireshark-gtk -i eth0 -k -w "${_WSLOG}" &

// __ running tcpdump

sudo tcpdump -pni eth0 -s65535 -G 3600 -w "${_ODIR}/tcpdump_%Y%m%d_%H%M%S.pcap" &

more options

jscher2000 said

Albretch said
I also need to save and keep the actual pages that are being downloaded by the browser (including the supporting css, javascript, images, ...)

I doubt that logging will give you a convenient format ...

That is why I am a bit more comfortable using more than 

one tool and keeping an eye on all of them. Also, gzip you can easily unzip

Modified by Albretch

more options

Using only "nsHttp:3" (LogLevel::Info) in the DE version with about:networking seems to give a log with the HTTP request and response headers and not that much extra data.