Hacker Public Radio

This show has been flagged as Clean by the host.

01



This is the fourth episode in a four part series on simple podcasting.







02 Introduction



In this episode we will discuss alternatives to Audacity when it comes to analyzing audio spectrums to find the sources of unwanted noise.



I previously promised some gratuitous hackery, and we will get into that in this episode.







03



Recall that with Audacity you first import the audio file, then select the part of the audio you wish to analyze (or ctrl-A for all), and then select analyze > plot spectrum.



This is in fact the only feature of Audacity that I know how to use. I am definitely not an audio expert.



I do however have some background in processing and analyzing other signals, so some of the basics are familiar to me.







04



We can accomplish the same thing that Audacity does in this instance provided we can do the following.



First, we need to get the data out of the audio file and into a form which we can import into other software.



Second, we need to perform certain mathematical operations on this data.



Finally, we need to be able to plot the results of these calculations on a chart.







--------------------







05 Fourier Transforms



First though, we need a bit of mathematical background.



What Audacity is doing when it shows a plot of frequency versus amplitude is that it is showing the results of a Fourier Transform.



A Fourier Transforms is a mathematical operation that converts the time domain into the frequency domain.



Any complex signal, audio or otherwise, can be broken down into a collection of sine waves of various frequencies.



For example, a simple square wave signal of say 100 hertz can be represented as a sine wave of frequency 100 hertz plus a collection of higher frequency sine waves which add together to give the sharp corners.







06



A Fourier Transform finds these sine waves and sorts them out into separate bins, with each bin representing an individual frequency or a collection of closely related frequencies, depending on how fine grained the sorting is.







07



This is exactly what we want when we are trying to figure out how to filter out noise.



Recall that earlier in this series we had to solve a problem with a high pitched background noise which was originating in my cheap microphone.



Analyzing this audio by frequency showed that it was a series of individual tones at 1 kHz intervals.



We were then able to use filters targeted at those frequencies to get rid of that noise.







08



There are several optimized versions of the Fourier Transform algorithm.



A very common one is the Fast Fourier Transform, common abbreviated to just "FFT".



This is so common that the term "FFT" is often used to simply mean any Fourier Transform even though this is not technically correct.







09



Typical FFT algorithms require that the number of data samples is exactly a power of two.



So the number of samples we need may be something like 4096, 8192, or 65536, to give a few random examples.



When we transform from the time domain to the frequency domain, each sample becomes a single frequency "bin". So the more samples we have, the finer the resolution we get in terms of frequency.







10



If we assume we are dealing with flac files recorded at a 44.1 kHz sample rate, that is, 44100 samples per second, then if we have 32768 samples, each "bin" represents slightly more than 1 hertz.



If we have 65536 samples, then each "bin" represents a fraction of a hertz.



For our purposes we will pick 65536 samples.



That means we need 1.48 seconds of data.



For simplicity's sake we will record at least 2 seconds of data and then just discard the samples that we don't need.







11



There is a further complication here. Fourier Transforms normally work with complex numbers.



Recall from your school days that as well as integers and real numbers there are complex numbers.



Each complex number consists of two parts, a real component and an imaginary component.



I won't go into the details of this, just accept that each sample needs to have two components.



Fortunately, if we don't have complex number data we can just set the imaginary component to zero and use that.







This is enough talking about the theory, let's get into the practical details.







--------------------











12 Extracting Data from Audio Files



First we will look at how to extract the data from the audio files.



Fortunately, one of the programs which we have already been using can do this.







To do this we will use Sox.



I am not aware of an equivalent feature in ffmpeg.







13



Sox calls itself "SoX - Sound eXchange, the Swiss Army knife of audio manipulation"



Sox is free software and is licensed under the GPLV2 or later.







In this case we want to use a feature which allows us to convert a binary audio signal file to a text data file.







To convert the file to text data we just give the output file a ".dat" file extension and Sox will do this for us.







14



Here is a command example.



sox inputfile.flac tdata.dat







15



This gives us a file in the following format, assuming this is a mono audio recording.







; Sample Rate 44100



; Channels 1



0 0.045471191406



2.2675737e-05 0.055023193359



4.5351474e-05 0.048217773438



6.8027211e-05 0.053192138672







etc.







The first line states the sample frequency



The second line states that the data is for channel 1.



The data starts on the third line.



Column 1 is the time in seconds.



Column 2 is the waveform data point.







16



To analyze the data we want a subset of these samples.



When we convert from the time domain to the frequency domain, our resolution will be determined by the number of samples. We would like therefore to have at least as many samples as the sampling rate.



We also want the samples size to be an even multiple of two.



The number of points we want to have is equal to the next even multiple of two above our chosen sampling rate, 44,100 Hz.



This number would be 65536.







17



To extract this data from the file we can do the following.



tail tdata.dat -n+3 | head -n65536 | awk '{printf "%s\n", $2}' > tdata.csv







18



We use tail to skip over the first three lines.



We use head to take the next 65536 lines and discard the rest.



We use awk to extract the second column which we will use as the real component.







We now have this data as a csv file in one column.







--------------------







19 Analyzing the Data



To analyze the data we need software which can calculate FFTs.



I will now show two examples of this, a very simple case using Libre Office Calc, and a more complex but more complete one using GNU Octave.







20 Using Libre Office



We can do fourier analysis and plot charts using Libre Office.







Take the csv file of data that we previously created.



For this example I used data from a recording of silence so that I could see what internal noise was being generated by the headset.



Open the csv file and import it into Libre Office Calc.







21



Now select all 65536 rows of column A.



The Fourier function will automatically fill the imaginary component with zeros if we don't provide an column of imaginary numbers, so we don't need to provide a column of zeros.



Then select Data > Statistics > Fourier Analysis.







22



A window will open allowing you to select various parameters.



For Results to:, enter "D1".



Grouped by Columns.



Select OK.







23



New data should now appear starting in cell D1.



The first line will say " Fourier Transform"



The second line will state the input range.



The third line will state "Real" in column D, and "Imaginary" in column E.



The data will start in row 4.







24



For our simple example we will ignore the imaginary data and just use the real data, which will form our Y component when we plot it on a chart.



We now need to create the X axis data.







25



Each cell is a "bin" of frequencies.



Each cell therefore represents (sample frequency) / (Number of samples) Hz.







26



To create the X axis data showing frequency, enter the following formula in to column C to the left of each D column number.



=((44100/65536) * (ROW() - 4)







27



We can now create an XY chart showing the frequency analysis.



You may need to exclude the first couple of dozen rows as very low frequency components which cannot be heard may otherwise overwhelm the data we are interested in.



Also, you only need the first half of the chart.



The FFT mirrors the data from the first half of the array into the second half.







28



Because characterizing a sine wave requires a minimum of 2 points, although we have a sample frequency of 44.1 kHz, we really only have sound waves up to a maximum of half that, or 22.05 kHz.



Create the chart with lines only.



If you followed the above instructions, you should see something resembling what we saw in Audacity, except with each bin more sharply defined.







29



In the data that I had from a recording of unfiltered headset noise, I could see a distinct noise spike every 1000 hertz.







30



However, we have taken several shortcuts.



First, the imaginary component of the data was ignored.



Second, the magnitude (that is, Y axis) has both positive and negative peaks.



Third, the data is not scaled to dB sound units, so we just have a relative measure.



However, that by itself is enough to tell us where the frequencies are that we need to construct filters to deal with.







31



We could refine this spreadsheet a bit more to deal with the above issues, but I think we have demonstrated the basic principle, and working with a spreadsheet can be a bit awkward.







However, if working with a spreadsheet is what you want to do, then you can add more columns and more formulae to improve on it.







--------------------







32 Other Analysis Software



I will go on to GNU Octave in a moment, but I want to get a few other alternatives out of the way first.



I won't go into any detail on them other than to point them out to people who want to have a go at trying these themselves.







33 Grace



There is math and plotting software called Grace.



This is free software, released under the GPL V2.



According to the documentation, it seems to have the features we need, including an FFT function.



However, I could not get it to work properly on Ubuntu 24.04.



I could not get it to load a data file and plot data.







34



The error messages were vague and unhelpful.



The file navigation system didn't work.



There was no obvious path to success, and if it isn't easy to use then there is no point to it.



This is fairly old software, designed for X Window and Motif.



I gave up on it as not suitable for this series as I am looking for some fairly low effort things for people to try themselves.



If someone else can get it to work on their PC, perhaps they could do an HPR episode on this themselves.







35 Command Line FFT Packages



There are several command line FFT packages.



They will read data from std in or from a file and output the FFT.



However, these are not packaged for Ubuntu and appear to be distributed as C source code which you would download and compile.



You can experiment with those if you wish, but I felt they were a bit out of scope for discussion here as I am looking at common tools that are ready to use.







36



Here are two examples.



One is



Command-line Fast Fourier Transform utility



https://github.com/gregfjohnson/fft







Another is



cli-fft



https://github.com/jonolafur/cli-fft







37



I have not tried these and cannot say whether they are any good or not.







Similarly, there are a number of FFT packages that are libraries for languages such as Python.



If you want to take the time to write a short program to go with them, you can create a dedicated FFT command line program.



However, I felt that this too was out of scope for what I was trying to do here.







38 Doing it the Hard Way



Hypothetically, it may be possible to write an FFT function in bash bc, which is the arbitrary precision calculator language which is part of the standard shell package.







I say hypothetically, because I have not tried it.



I think it would be an interesting challenge, but I don't have the time at the moment to try it.



If anyone feels motivated to give it a try, they're welcome to give it a go and then do a podcast episode on it.







--------------------







39 GNU Octave







We have seen that as well as using features built into Audacity to analyze the audio spectrum to see the frequencies of undesired noises, we were able to do the same using a Libre Office spreadsheet.







40



Now we'll look at another bit of software, GNU Octave. GNU Octave is free software, licensed under the GPL V3 or later. It is a mathematical scripting language, very similar to Matlab. People use it for mathematical, engineering, and scientific work. It can be found in most Linux distros and is available for some other operating systems as well.







41



Octave has two features built in that we need for our purposes. It does FFTs, and it has a plotting system built in to produce graphs.







--------------------







42



We will take the same audio test file that we used with Audacity and Libre Office and use it here as well.







The bash script to convert the flac file to text data is essentially the same, with the exception that file extension on the output file as is ".txt" instead of ".csv". This latter change was an arbitrary decision on my part.







43



As a quick review, this bash script uses sox to convert a flac file to a text ".dat" file.



Then it uses tail, head, and awk to extract the first 65536 rows of data, skipping over the header information and ignoring the first column of time data.



This script will be in the show notes.







--------------------







#!/bin/bash







# This version is for use with the GNU Octave script.







sox hsnoisemono.flac hsnoisemono.dat







tail hsnoisemono.dat -n+3 | head -n65536 | awk '{printf "%s\n", $2}' > hsnoisemono.txt







--------------------







44



We now have a 1.1 MB file containing 65536 samples of data in text format.







Now the next thing we need to do is to create a short Octave script file.



I will just give a brief overview of the script here, the full script will be in the show notes.







45



I put the script in a file called "octavespectrum.m". I have never used Octave before now, but the convention seems to be to give the script a ".m" ending.







The "she-bang" line is "#!/usr/bin/env octave". If you make the file executable you can run it like any other script, or you can type "octave" and then the name of the script to run.







46



I won't read out the script in detail, as that would be too hard to following along in a podcast.







However, I pass several arguments to the script including the name of the data file, and then two integers that I use to limit the display area in the Y and X axes so I can have the chart focus on the areas of interest that I want to see.



I also pass a string containing the name of the graphic file that I want the chart exported to.



This was an arbitrary decision on my part and you can just hard code these values in if that is what you want to do.







47



The arguments are accessed by calling the "args()" function, which returns an array of strings.







Next, it reads in the specified file using the "dlmread()" function. This reads all of the data into an array.







48



Next, it performs a hamming windowing function on the data. I'll explain that briefly.



It is standard practice when doing FFT signal processing to "window" the signal.



Since the signal sample is of finite length, it will stop at each end of the array.







49



Unless you were lucky enough for this to happen exactly at a zero crossing, this would produced an abrupt transition in the data which looks like "noise" to the FFT.



The solution is to taper the signal off gradually towards the ends so that when it gets cut off the signal is fairly small at that point anyway.



There are a variety of different windowing functions, but "hamming" seems to be the most commonly used.







50



Next, it does an FFT using the "fft()" function.







51



This gives us real and imaginary outputs.



These are combined by summing the squares of each corresponding real and imaginary element and then taking the square root of each and storing that in a new array. This gives a single array of the same length as the originals, but combining the two output components.



If anyone wants to tell me that this isn't how things are done in the audio world, they're welcome to make an HPR episode telling us all the right way to do things.







52



Then it does some scaling and selection of subsets of data so we get the X axis in hertz and just the number of samples that we wish to look at.







If you are looking at the script, the thing to keep in mind is that Octave will work on entire arrays of data in a single operation. You don't need to write explicit loops for this. The looping is handled implicitly as part of the syntax.







53



It also does various other things that make the chart easier to read. The comments in the script describe these in more detail. Since this is a script it's easier to add these sorts of refinements than is the case for a spreadsheet so I have made the effort to add them.







Finally it calls the "plot()" function.



If an output graphics file name was provided, it also creates a PNG file containing the same image using the "saveas" function.







54



We now see the chart, and it looks more or less as expected.



However, this chart is interactive.



You can zoom and pan the data, something that you can't do with either Audacity or Libre Office.







The chart window doesn't have a function for exporting the resulting chart to a "png" file, it will only save to an ".ofig" file. The ofig file is not a standard graphics file, it is a serialization of the chart data that can only be looked at using the Octave chart viewer.







55



Alternatively, you can just take a screenshot of the chart after you have interactively zoomed and panned to a point of interest.







At the bottom left of the chart window is a pair of x-y coordinates which tell you the current position of the mouse pointer in chart units.



This is very handy as it can be used to get the exact (or close to exact) frequency of each noise spike.







56



The Y axis is not scaled in any particular units such as dB, as I'm not sure how to do that according to audio industry conventions.



On the other hand, I'm not sure that it's really necessary, as I don't know what dB means in tangible terms anyway.



It does show relative sizes, so it helps to determine whether you have one noise frequency or multiple frequencies to worry about.







57



If anyone is familiar with how to scale the raw data from a flac file as exported by Sox into dB units according to audio industry convention, then they are welcome to create an HPR episode telling us how to do it.







--------------------







58 Comments on GNU Octave







I had never used GNU Octave before this, although I had heard of it and it is quite a significant piece of software for a specific segment of users.







59



The syntax is a bit odd especially in how it deals with array operations, but I was able to google various examples and answers to eventually get this working.



A few other peculiarities are that it uses the percent "%" character to denote a comment, and leaving out the semi-colon at the end of the line causes it to print the answer to the console after executing the statement.







60



The GNU Octave solution was harder to get working than the Libre Office method.



However, once it was working it is easier to use repeatedly.







If I were to want to automatically generate audio files with different filtering or other options and wanted to script the creation of a large number of images showing the results, this would be the way to do it.







61



When your run the Octave script you may get a warning which says something like



"QSocketNotifier: Can only be used with threads started with QThread".



This is apparently a routine warning message from the Qt graphics system which has no real significance in this context and can be ignored for our purposes.







--------------------







62



We now have a bash script which will use sox to extract the data from a flac file, and a GNU Octave script which can be used to display the resulting frequency spectrum.







This does more or less the same thing as "Plot Spectrum" does in Audacity, but allows for zooming and panning to get a more detailed look at the data.







63



However it doesn't give you an absolute reading of the sound levels in dB, something that Audacity does provide.







What I wanted it for though was to find the frequencies of the audible noise in the signal, something that it does quite well.











--------------------







#!/usr/bin/env octave







% Perform an FFT on the data in a file and plot the results.







% ======================================================================







% The sampling frequency. This must be changed to accommodate the



% actual sampling frequency if it was something else.



samplefreq = 44100;







% Thickness of line on plot.



linewidth = 2;







% ======================================================================







% The name of the data file is passed as a argument.



args = argv();



if length(args) < 3



quit



endif











% File name.



fname = args{1};



% Clip the peak values.



peakclip = str2double(args{2});



% How much data to show, in kHz.



rbound = str2double(args{3}) * 1000;



% The optional file name to save a chart image to.



if length(args) > 3



chartfile = args{4};



else



chartfile = "";



endif











% ======================================================================



% Read the data in from the file.



sampledata = dlmread(fname);



% Number of samples.



samplecount = length(sampledata);







% ======================================================================







% Window the data. This helps deal with the discontinuity of data at



% each end of the array and the effects this has on introducing apparent



% noise into the signal.



windoweddata = (hamming(samplecount) .* sampledata);







% ======================================================================







% Do the actual FFT.



fftresults = fft(windoweddata);







% Get real component.



r = real(fftresults);



% Get the imaginary component.



i = imag(fftresults);







% Combine the real and imaginary. In order to square each element of each



% array, we must use the ".^" operator, not just "^".



rfft = sqrt(r.^2 + i.^2);



realfft = rfft(1:samplecount);







% ======================================================================







% Scale factor for frequency.



fscale = samplefreq / samplecount;



% X axis scale, scaled to frequency.



f = (0:samplefreq/2) * fscale;



% Take a subset of the data if specified. rbound has to be re-scaled



% from kHz to array increments.



freq = f(1:min(rbound / fscale,length(f)));











% y axis. We take the absolute value and then limit (clip) the peaks



% so that a few large peaks don't obscure the smaller ones.



mag = min(abs(realfft(1: length(freq))), peakclip);











% Plot the results.



figure;



whandle = plot(freq, mag, 'LineWidth', linewidth);



title(["Audio Spectrum of ", fname]);



xlabel("Frequency (Hz)");



ylabel("Unscaled Magnitude");



grid on;







% If the appropriate optional argument was specified, save the chart



% to a file of that name.



if length(chartfile) > 4



saveas(gcf, chartfile, "png");



endif







% Need this so the plot window stays open.



waitfor(whandle);







% ======================================================================







--------------------







This is the shell script used with the above Octave script.



The arguments are



1 - the file name for the input data file.



2 - The value to clip the peaks at.



3 - The upper frequency bound in kHz.



4 - The output graphics file name.







#!/bin/bash







octave octavespectrum.m hsnoisemono.txt 10 12 hsnoisemono.png











--------------------







64 Episode Conclusion



In this episode we covered the following topics.



What Fourier transforms are.



Extracting data from audio files using Sox.



Analyzing the data using Libre Office.



Analyzing the data using GNU Octave.



And, several alternative analysis methods.











65 Series Conclusion



This is the end of a four part series on simple podcasting.







In the first episode, we covered a simple podcast recording method. This first episode is all you really need to make a podcast.







66



In the second episode we covered basic filtering and a few other simple topics. The methods discussed in that episode provide basic improvements to your audio if you feel the need for it.







67



In the third episode we covered how to analyze audio noise problems using Audacity and additional filtering techniques to deal with specific problems that we may find.



We also covered command line recording, playback, and getting information about an audio recording.







68



In the fourth episode we engaged in a bit of gratuitous hackery for the fun of it and showed how to use alternative software methods to analyze audio signals.







69



I hope that this series has been both useful and entertaining and that you will use the knowledge gained here to create and submit your own HPR podcast episodes.







--------------------



--------------------





Provide feedback on this episode.

This show has been flagged as Clean by the host.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.


In UNIX Curio #4 (
HPR episode 4617
), I teased the subject of file compression. Today I'm circling back to that.



The history of data compression goes back at least to the 1970s, and in contexts outside UNIX and computers, probably even earlier. Somehow, it is refreshing to learn that humans have always struggled to have enough storage space to keep all the data they want to hang on to. One way around this limitation is to use some form of compression.



I am only going to dive into
lossless
compression for this episode—that is, a compression method that can be reversed and will spit out the original data bit for bit. Lossy compression methods also have their places: you might be familiar with their use for audio (such as Ogg Vorbis or MP3); it's also used for images (such as JPEG). Lossy compression allows some of the original data to be thrown away, resulting in a smaller file than is possible with lossless compression, but the intent is for the result to still sound or look "good enough" to a human observer. Also, I am going to limit my discussion to generic methods used for many types of data; while FLAC does lossless compression, it is specifically designed just for audio.



I should make clear that I have never studied computer science or information theory, so this episode will not get into the science behind various types of compression algorithms and how they differ. But in general, these methods take advantage of the fact that many types of data have recurring patterns. English text mostly consists of words that often re-appear many times—source code similarly has keywords and variable names that recur. Compression is accomplished by representing a piece of data that occurs multiple times with a symbol that is shorter in length.



The first compression program in the UNIX world I could find is called


pack




, from 1978


1
. It was shortly followed in 1979 by a similar program called


compact




2
. Both of these used a technique called Huffman coding, but with some differences between them. Files compressed with
pack
were given a
.z
extension and
compact
gave filenames a
.C
extension. Roughly every five or ten years after this, a new program would come along and achieve lasting popularity.



There were, and still are, two opposing forces facing any new form of compression. Working in favor was the advantages it provided—first among these was achieving a better compression ratio, but performance improvements such as speed or reduced memory usage could also be compelling. The force against any new method was the fact that it was not yet widely supported—it doesn't much help to have a smaller file if the people you share it with cannot decompress it.



The next major advance in compression arose out of three scientific papers: two in 1977 and 1978 by Abraham Lempel and Jacob Ziv (called LZ77 and LZ78), and one by Terry Welch in 1984 which built on LZ78. This last method is typically referred to as LZW. Our UNIX Curio for today is a
program called




compress




3
that implements the LZW method. Files compressed this way are named with the extension
.Z
. I had always assumed that this was to honor Jacob Ziv, but now that I've researched the history, it seems more likely to be a follow-on from how files compressed by
pack
were named. Since
pack
did not use any of the Lempel-Ziv methods, I would guess that it used
.z
because that wasn't already taken by anything else, but that's pure speculation.



I do recall encountering
.Z
files in the wild, but feel certain that hasn't happened in the last 25 years, maybe longer. If you need to expand one of these,


uncompress




4
is the program to use (
GNU's




gunzip




can also handle them


5
). However, there was a serious problem that arose with the LZ78 and LZW compression methods. Both of them were patented, and the owner became aggressive in seeking payment from developers and users. The
compress
utility was developed within two months of the publication of Welch's 1984 paper and was included in Bell Laboratories' Eighth Edition UNIX before these shakedowns started. The paper did not disclose that a patent had been filed, and apparently Spencer Thomas and the other developers of
compress
were unaware of it. The utility became popular for a while, and was even standardized by POSIX, but people moved away from LZW once the legal threats started.



Another important advance came in 1991 and was called the DEFLATE compression method. It combined the un-patented LZ77 method with Huffman coding to achieve a similar level of compression as LZW (actually, often better) without the legal trouble. DEFLATE was developed for
PKZIP
and was soon adopted by the GNU project's
gzip
compressor. While Phil Katz (the "PK" in
PKZIP
) patented one way of implementing the DEFLATE method,
it was possible to write a compressor and decompressor without infringing


6
; also, he apparently
never tried to enforce the patent


7
.



As I mentioned in UNIX Curio #4, .zip is both an archive
and
a compression format. Each archive member can be compressed with one of several possible methods (or stored without compression). Unlike a
tar
file where compression can be applied to the entire archive, in .zip each archive member is compressed individually. This often means a .zip file will be slightly bigger than a
tar
file with the same contents compressed with
gzip
, because the .zip format cannot take advantage of duplication that occurs among more than one member of the archive. The vast majority of .zip files use only the DEFLATE and uncompressed storage methods and these are the only options if you want to follow the profile standardized in ISO/IEC 21320-1. Actually, since they both use DEFLATE,
gzip
is able to extract a .zip file in the special case where it only holds one member compressed with that method.



From the 1990s onward, people paid significant attention to avoiding patent landmines, so only methods that didn't have that problem became broadly popular. While the patents on LZ78 and LZW have since expired, I feel like their most successful legacy was in discouraging people from using those methods, leading to DEFLATE taking the popularity crown.



The next step came in 1996 and 1997 with the development of
bzip
and
bzip2
by Julian Seward. The original method was quickly followed by
bzip2
, which was the version that achieved true popularity. They use the Burrows-Wheeler transform, which does not itself compress data but re-arranges it to make it more compressible;
this is combined with other techniques


8
. (At least, that's my understanding. I told you, I'm not up on information theory.) This provides a significant reduction in the compressed size of the data compared to earlier methods—however, it is slower than DEFLATE both during compression and decompression.



Separate projects have developed parallel versions of
gzip
and
bzip2
that can take advantage of multi-processor machines, but the original utilities run single-threaded.



Another five years later, in 2001, Igor Pavlov added the Lempel-Ziv-Markov chain algorithm (LZMA), an enhancement to LZ77, to his 7-Zip compression tool. This was followed a few years later by LZMA2, a container format that allowed for LZMA compression to be split between multiple threads. Broad LZMA2 support came to the UNIX world in 2009 with the


xz




utility


9
. It offers roughly similar compression ratios to
bzip2
, though it can be better or worse depending on the data to be compressed. While compression generally takes even longer than
bzip2
, decompression is significantly faster (though still not as fast as
gzip
). The Linux kernel relatively quickly supported
booting from xz-compressed images


10
because it was a good match for that use case—compression, the time-consuming activity, only has to be done once while the more frequent decompression during boot happens relatively fast.



The last method I will cover is
Zstandard


11
, often written as
zstd
. This came about in 2015, and is another variation on LZ77 that uses finite-state entropy (which means nothing to me, but you might understand it). It performs about as well as DEFLATE in terms of compression ratios, but is much faster both when compressing and decompressing data. I should say that these statements are true with the typical default settings—depending on the compression level selected, it can compress more slowly, but compress the data smaller. However, decompression is always speedier than DEFLATE. This makes it attractive for some uses, and it is heavily promoted by Meta/Facebook, where Yann Collet developed it. For example, shipping large amounts of actively-used data between machines in a data center can go more quickly when the size is reduced; however, if the compression and decompression steps take too long that benefit is lost. A speedy method can be valuable even if it doesn't result in the greatest reduction in size. This use case stands in contrast to, say, a compressed backup file which might only be accessed in a disaster recovery scenario or never accessed at all, making size more important than speed.



Both the
xz
and
zstd
utilities have some built-in support for multi-threading, but the default is to run in a single thread. While
xz
can use multiple threads for decompression (but only if the file was compressed in multi-thread mode), the reference
zstd
utility can only use more than one thread for compression, not decompression.



There are many other methods of lossless compression that have been developed over the decades, but I believe these are the ones you are most likely to encounter in the world of UNIX-like systems. This is a personal opinion, and others might choose a different set. As mentioned, it can be tough for a new method to gain popularity and 35-year-old DEFLATE is still probably the most commonly used despite not being the fastest or offering the greatest reduction in size. Even systems like FreeBSD, NetBSD, and OpenBSD that do not like to include GNU tools supported it by developing their own version of
gzip
based on the permissively-licensed
zlib
library.



Technically, the LZW method used by the
compress
utility is still standardized by POSIX, so one might expect it to have the widest support. However, aggressive patent enforcement discouraged adoption, especially by Free and Open Source Software systems—even though the patent has expired, it is still out of favor compared to DEFLATE. For this reason, I feel justified in calling it a curio.



References:







Eighth Edition UNIX pack.c
https://www.tuhs.org/cgi-bin/utree.pl?file=V8/usr/src/cmd/pack/pack.c





2.9BSD compact.c
https://www.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/ucb/compact/compact.c





Compress specification
https://pubs.opengroup.org/onlinepubs/009695399/utilities/compress.html





Uncompress specification
https://pubs.opengroup.org/onlinepubs/009695399/utilities/uncompress.html





GNU Gzip manual
https://www.gnu.org/software/gzip/manual/gzip.html





RFC 1951: DEFLATE Compressed Data Format Specification version 1.3
https://tools.ietf.org/html/rfc1951





History of Lossless Data Compression Algorithms: The Rise of Deflate
https://ethw.org/History_of_Lossless_Data_Compression_Algorithms#The_Rise_of_Deflate





bzip2
https://en.wikipedia.org/wiki/Bzip2





XZ Utils
https://en.wikipedia.org/wiki/XZ_Utils





2.6.38 merge window part 2
https://lwn.net/Articles/423541/





zstd
https://en.wikipedia.org/wiki/Zstd







Appendix




The table below demonstrates the results of compressing different types of data using tools described in this episode. While not totally rigorous, I did run each compression and decompression multiple times to ensure I was getting consistent results. The laptop I used has an Intel Core i5-6200U CPU running at 2.30GHz, and the system had at least 5 GB of free memory for each run. While this processor has two cores and can run four simultaneous threads, all utilities were run single-threaded.



The term "best" means the highest level of compression available (the exact level used is shown). For
bzip2
, the default
is
the best. For
zstd
, "best" is -19, which is the highest "normal" level, but "ultra" levels that are even higher also exist. Ratios are the percentage of the original size that the file was reduced to (other sources might instead express the compression ratio as the
reduction
in size achieved). In all results, smaller numbers are better.



┌────────────────────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐
│ │ gzip │ gzip │ bzip2 │ xz │ xz │ zstd │ zstd │
│ │(default -6) │ (best -9) │ (-9) │(default -6) │ (best -9) │(default -3) │ (best -19) │
├──────────────┬─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Size (ratio) │ 22,036,508 │ 21,891,623 │ 15,795,698 │ 13,487,768 │ 12,938,464 │ 20,454,657 │ 13,709,078 │
│ │ │ (24%) │ (24%) │ (17%) │ (15%) │ (14%) │ (23%) │ (15%) │
│English Text ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(90,532,092 │Compression │ 4.8s │ 7.6s │ 8.5s │ 49.8s │ 58.8s │ 0.6s │ 65.2s │
│bytes │time │ │ │ │ │ │ │ │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Decompression│ 0.7s │ 0.8s │ 3.7s │ 1.2s │ 1.2s │ 0.4s │ 0.4s │
│ │time │ │ │ │ │ │ │ │
├──────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Size (ratio) │ 125,291,122 │ 124,189,544 │ 98,016,512 │ 84,882,492 │ 81,954,344 │ 120,604,855 │ 87,298,645 │
│ │ │ (21%) │ (21%) │ (17%) │ (14%) │ (14%) │ (20%) │ (15%) │
│Source Code ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(590,008,320 │Compression │ 22.0s │ 39.3s │ 54.8s │ 241s │ 298s │ 3.7s │ 348s │
│bytes │time │ │ │ │ │ │ │ │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Decompression│ 5.1s │ 5.1s │ 20.3s │ 8.1s │ 7.8s │ 2.4s │ 2.4s │
│ │time │ │ │ │ │ │ │ │
├──────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Size (ratio) │ 32,830,905 │ 32,371,241 │ 26,856,579 │ 20,717,288 │ 20,352,880 │ 28,538,810 │ 23,154,582 │
│ │ │ (19%) │ (19%) │ (16%) │ (12%) │ (12%) │ (17%) │ (13%) │
│Binary Program├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(171,972,264 │Compression │ 6.4s │ 22.4s │ 18.6s │ 62.2s │ 67.8s │ 0.8s │ 111s │
│bytes │time │ │ │ │ │ │ │ │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Decompression│ 1.5s │ 1.5s │ 5.6s │ 2.3s │ 2.3s │ 0.7s │ 0.7s │
│ │time │ │ │ │ │ │ │ │
├──────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Size (ratio) │ 146,397,772 │ 146,397,757 │ 144,485,451 │ 131,950,232 │ 130,926,780 │ 147,154,979 │ 145,703,840 │
│ │ │ (89%) │ (89%) │ (88%) │ (80%) │ (80%) │ (90%) │ (89%) │
│WAVE Audio ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(164,396,302 │Compression │ 9.2s │ 9.2s │ 25.1s │ 70.4s │ 97.7s │ 0.7s │ 58.3s │
│bytes │time │ │ │ │ │ │ │ │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │Decompression│ 2.0s │ 2.0s │ 13.5s │ 12.2s │ 12.1s │ 0.6s │ 0.8s │
│ │time │ │ │ │ │ │ │ │
├──────────────┴─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│ │ gzip │ gzip │ bzip2 │ xz │ xz │ zstd │ zstd │
│ │(default -6) │ (best -9) │ (-9) │(default -6) │ (best -9) │(default -3) │ (best -19) │
└────────────────────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘





English text consists of Titles 1 through 10 of the 2020 U.S.
Code of Federal Regulations
.



Source code consists of a
tar
file containing the Linux kernel source, version 4.0.



Binary program consists of an ELF-format executable of the
pandoc
application, version 2.17.1.1 found on Debian 12.



Audio consists of a 24-bit Signed Integer PCM WAVE file with 2 channels at 44.1kHz, about 10:21 in length. For comparison, the audio-specific
flac
lossless compression utility reduced this file to 97,962,711 bytes (60%) in 2.6 seconds at the default (-5) level and to 97,714,876 bytes (59%) in 5.4 seconds at the highest (-8) level.







Provide feedback on this episode.

This show has been flagged as Explicit by the host.



Games Mentioned



Tile Survive - https://tilesurvivegame.com/en

Monopoly Go - https://apps.apple.com/us/app/monopoly-go/id1621328561

Arrows Go - https://apps.apple.com/us/app/arrows-go-arrow-puzzles/id6758326278



Apologies, Lee's mic is a bit metallic due to filtering out background noise from a fan


Provide feedback on this episode.

This show has been flagged as Explicit by the host.


WARNING AI GENERATED NOTES AHEAD YMMW








Here is a summary of the recorded training session regarding Android hacking from Hacker Public Radio, including web references for the main topics discussed.








Overview




The recording features a security consultant performing a live assessment of an Android application. The consultant uses a custom tool suite called "Jamboree" and various other utilities to test a location-sharing and vehicle management app. The session highlights the increasing complexity of mobile app security, specifically dealing with SSL pinning, encrypted traffic, and anti-tampering mechanisms

1

.




Environment and Tools




The assessment is conducted on a rooted Android emulator. The speaker utilizes several tools to set up the environment and intercept traffic:







Jamboree

: A custom automation tool developed by the speaker over six years to handle rooting, proxy setup, and app installation within minutes

1

.





Burp Suite

: The primary interception proxy used to analyze traffic between the app and the production server

1

.





Frida

: Used to bypass anti-root detection and SSL pinning

1

.





Ghidra

: A decompiler used to analyze the app's code, specifically helpful for patching the Flutter-based application

1

.





Android Debug Bridge (ADB)

: Used for troubleshooting, debugging, and analyzing logs (

logcat

) to extract user IDs and location data

1

.






Technical Challenges: SSL Pinning and Flutter




The target application is built using Flutter and implements rigorous security controls, including SSL pinning, which prevents standard Man-in-the-Middle (MitM) attacks. The app's HTTP client ignores system and user-installed certificates, and it does not respect device Wi-Fi proxy settings

1

.




To overcome this:







Traffic Redirection

: The speaker uses

iptables

commands to force all HTTP and HTTPS traffic through the proxy's IP address at the network layer, bypassing the app's proxy ignorance

1

.





Patching with AI

: The speaker leverages AI (specifically mentioning Claude and access to "Kuro") to assist in patching the APK. The AI helped navigate Ghidra and generate Python scripts to bypass the app's protections, allowing the modified APK to trust the auditor's certificate

1

.





Frida Scripts

: "Frida anti-root SSL pinning" scripts are executed to further mitigate detection mechanisms

1

.






Key Vulnerabilities Identified





1. Geolocation Spoofing





The consultant successfully spoofed the device's GPS location using emulator settings (e.g., setting the location to Puerto Rico or Costa Rica). The application accepted this falsified location data as valid, indicating a lack of server-side verification for location origin

1

.





2. Insecure Direct Object Reference (IDOR) / Broken Access Control





The most critical finding involves the app's user tracking feature.






The consultant discovered that the API allows querying a user's location via a

user_id

.




By intercepting traffic and analyzing

adb logcat

logs, the consultant extracted their own

user_id

and the

user_id

of a second test account

1

.




While authenticated as one user, the consultant was able to send a request substituting the

user_id

with the target's ID. The server responded with the target's GPS coordinates. This confirms that an authenticated user can track any other user's real-time location if they possess the target's ID

1

.




Proof of concept was created by copying the request as a

curl

command to demonstrate the exploit

1

.







3. Potential Information Disclosure





The consultant began testing a feature that allows users to add vehicles by license plate. The concern is that querying a license plate might return excessive PII (Personally Identifiable Information), such as VIN numbers or registration details, beyond what the UI strictly requires (least privilege issue)

1

.





4. Access Control (Calendar Feature)





The consultant tested whether calendar events could be accessed by switching

user_id

parameters. This test resulted in a "401 Unauthorized" error, indicating that this specific endpoint had proper access control in place

1

.




Web References and Resources




Below are references for the main tools and concepts discussed in the training:







Hacker Public Radio

:

https://hackerpublicradio.org/






Burp Suite (Web Security Testing)

:

https://portswigger.net/burp






Frida (Dynamic Instrumentation Toolkit)

:

https://frida.re/






Ghidra (Software Reverse Engineering)

:

https://ghidra-sre.org/






Android Debug Bridge (ADB)

:

https://developer.android.com/tools/adb






OWASP Mobile Top 10

:

https://owasp.org/www-project-mobile-top-10/






OWASP Testing for Insecure Direct Object References (IDOR)

:

https://owasp.org/www-project-web-security-testing-guide/v42/4-Web_Application_Security_Testing/04-Authorization_Testing/04.1-Testing_for_Insecure_Direct_Object_References






Flutter (UI Toolkit)

:

https://flutter.dev/








Provide feedback on this episode.

This show has been flagged as Clean by the host.


Hi this is your host, Archer72 for Hacker Public Radio.






In this episode I share some of my findings about a problem with
the Newsboat naming of the HPR feeds,


which was brought up in comments about my Newsboat show, HPR4424.







hpr4424: How I use Newsboat for Podcasts: comment #6 :
download-filename-format for HPR podcasts






Ken already had some findings of his own about the
ccdn.php extension in the feed.







hpr4424: comment #10 : Summary of findings






I thought that this might be able to be fixed on an invididual
basis, and set out to ask Claude.ai a few questions.






But first, some colaboration from Dave Morriss about a good
renaming format. This was definitely more on Dave’s side than
mine, but came up with this.


You can tell Dave’s handywork from the short variable names, which
stems from his extensive experience on Unix type machines in the
University days.






exif-rename-hpr-dave.sh





#!/bin/bash

URL="$(cat /tmp/hpr-url.txt)"
echo "DEBUG URL: $URL" >> /tmp/hpr-debug.log

AUDIO_URL="$(curl -s "$URL" | grep -Eo 'https?://[^"]*\.(ogg|mp3)' | head -1)"
echo "DEBUG AUDIO: $AUDIO_URL" >> /tmp/hpr-debug.log

if [[ -z "$AUDIO_URL" ]]; then
echo "ERROR: Could not find audio URL from: $URL" >> /tmp/hpr-debug.log
exit 1
fi

# Changed destination to HPR-queue
DEST=~/podcasts/hub.hackerpublicradio.org/HPR-queue/

# Record files present before download
BEFORE="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"

wget -nc --content-disposition -P "$DEST" "$AUDIO_URL"

cd "$DEST"

# Record filename just downloaded (new file not in BEFORE)
AFTER="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
DOWNLOADED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER"))"
echo "DEBUG DOWNLOADED: $DOWNLOADED" >> /tmp/hpr-debug.log

~/bin/exif-rename-hpr-dave.sh

# Find renamed file — newest file that wasn't in BEFORE
AFTER_RENAME="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
RENAMED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER_RENAME"))"
echo "DEBUG RENAMED: $RENAMED" >> /tmp/hpr-debug.log

if [[ -n "$RENAMED" ]]; then
echo "\"$AUDIO_URL\" \"$RENAMED\" downloaded" >> ~/.local/share/newsboat/queue
else
echo "WARN: Could not determine renamed file" >> /tmp/hpr-debug.log
echo "\"$AUDIO_URL\" \"$DOWNLOADED\" downloaded" >> ~/.local/share/newsboat/queue
fi





At first the question was about something simple. The input was a
query on one of the lines from Kevie’s







hpr4398 :: Command line fun: downloading a podcast






Particularly, the section on To get the latest episode of TuxJam






wget
curl https://tuxjam.otherside.network/feed/podcast/ | grep -o
'https*://[^"]*ogg' | head -1






Which I re-wrote to:





wget -nc -P ~/podcasts/TuxJam $(curl https://tuxjam.otherside.network/feed/ogg | grep -Eo 'https*://[^"]*ogg' | sort -u | xargs | head -1)





The reason for $() instead of backticks to enclose a command was
that the former was being deprecated.







GNU Bash Reference Manual - 3.5.4 Command Substitution






-nc –no-clobber is to not re-download a podcast -P specifies
download directory






I went on a different direction than downloading TuxJam and asked
to download the last 2 hpr shows, but
head -2 did not work as expected. This turned out to be
an issue with the placement of
xargs joining all URLs and passing them to
wget all at once.






Original:





wget -nc --content-disposition -P ~/podcasts/hub.hackerpublicradio.org/HPR-newsboat-test/ $(curl [http://hackerpublicradio.org/hpr\_ogg\_rss.php](http://hackerpublicradio.org/hpr_ogg_rss.php) | grep -Eo 'https\*://\[^"\]\*ogg' | sort -u | xargs | head -2)





New:





curl http://hackerpublicradio.org/hpr_ogg_rss.php | grep -Eo 'https?://[^"]*\.ogg' | sort -u | head -2 | xargs wget -nc --content-disposition -P ~/podcasts/hub.hackerpublicradio.org/HPR-newsboat-test/


Key fixes:







Removed the premature
xargs before
head -2
, so
head actually limits the list to 2 URLs



Moved
xargs wget ... to the end, after the list is already
trimmed



Cleaned up
https* →
https? (the original would also match
httpssss
, etc.)







Now I wanted the downloaded file to go the the queue file, located
in
~/.local/share/newsboat/queue



After several iterations in
Claude
, it was determined that the audio URL was not being expanded by
the
%u in the download macro.






The solution was to add a /tmp file to hold the actual audio URL





macro d set browser "echo %u > /tmp/hpr-url.txt && ~/bin/download-and-rename-hpr.sh"; open-in-browser ; set browser "your-normal-browser"





A few Claude questions later… After asking to add a function to
the macro in order to add the resulting downloaded file to the
queue, the file name hpr1234.ogg was being added to the queue
instead of the renamed file.






Two things to fix:







The renamed file isn’t being found because
ls -t runs before the rename has fully settled, or the
glob isn’t matching the new filename format (which includes
spaces and semicolons)



The existing queue entries show the correct format:
"url" "path" downloaded — we need to match that, with
downloaded status and the full renamed path including
extension







Exif rename script





#!/bin/bash
# ~/bin/download-and-rename-hpr.sh

URL="$(cat /tmp/hpr-url.txt)"
echo "DEBUG URL: $URL" >> /tmp/hpr-debug.log

AUDIO_URL="$(curl -s "$URL" | grep -Eo 'https?://[^"]*\.(ogg|mp3)' | head -1)"
echo "DEBUG AUDIO: $AUDIO_URL" >> /tmp/hpr-debug.log

if [[ -z "$AUDIO_URL" ]]; then
echo "ERROR: Could not find audio URL from: $URL" >> /tmp/hpr-debug.log
exit 1
fi

DEST=~/podcasts/hub.hackerpublicradio.org/HPR-newsboat-test/

# Record files present before download
BEFORE="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"

wget -nc --content-disposition -P "$DEST" "$AUDIO_URL"

cd "$DEST"

# Record filename just downloaded (new file not in BEFORE)
AFTER="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
DOWNLOADED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER"))"
echo "DEBUG DOWNLOADED: $DOWNLOADED" >> /tmp/hpr-debug.log

~/bin/exif-rename-hpr-dave.sh

# Find renamed file — newest file that wasn't in BEFORE
AFTER_RENAME="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
RENAMED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER_RENAME"))"
echo "DEBUG RENAMED: $RENAMED" >> /tmp/hpr-debug.log

if [[ -n "$RENAMED" ]]; then
echo "\"$AUDIO_URL\" \"$RENAMED\" downloaded" >> ~/.local/share/newsboat/queue
else
echo "WARN: Could not determine renamed file" >> /tmp/hpr-debug.log
echo "\"$AUDIO_URL\" \"$DOWNLOADED\" downloaded" >> ~/.local/share/newsboat/queue
fi





Using
comm -13 on sorted before/after file lists is more
reliable than
ls -t — it finds exactly which file is new regardless of
timing. Check
/tmp/hpr-debug.log after the next run to see what
DOWNLOADED and
RENAMED show.







GNU Coreutils - Comm invocation






Newsboat config snippet






key changes





download-filename-format "%t.%e"
macro d set browser "echo %u > /tmp/hpr-url.txt && ~/bin/download-and-rename-hpr.sh"; open-in-browser ; set browser lynx



Provide feedback on this episode.