Foundations of Amateur Radio

Foundations of Amateur Radio

While spending some quality time discovering what I don't know about GNU Radio, I explored the notion of attempting to at least understand a little more about how an FM signal works. Depending on your background, the letters FM mean different things. In amateur radio it's a way to encode information, generally audio, using something called frequency modulation. Outside the hobby, the letters point at commercial broadcast radio.

While the two are related, they're not the same thing.

In amateur radio use, FM is a single channel of mono-audio, however, in commercial broadcast radio, there's a whole lot more going on, interesting because it gives you ready-made access to a composite signal that's just complicated enough to be challenging without being so complex that you need to spend hours on understanding the thing.

In essence, a commercial FM broadcast signal is multiple channels encoded in a specific and documented way. This is helpful, since you can compare the documentation against ready made examples and replicate the process for yourself.

In case you're new here, I'm in the process of building a radio system, in software, using GNU Radio in a project called Bald Yak. Specifically, the Bald Yak project aims to create a modular, bidirectional and distributed signal processing and control system that leverages GNU Radio. It's called Bald Yak because by the time I'm done, the Yak is likely well and truly shaved.

One of the easy things to forget when you're using GNU Radio Companion, is that the blocks you're connecting together on the screen into a flowgraph actually represent software, generated when you either build or run the flowgraph. This code is currently generated in either Python or C++, making me wonder, what does the code look like, and more specifically, what code would be needed to decode FM?

It turns out that an old friend, the PySDR.org website has a whole chapter dedicated to this process. Chapter 18, the End-to-End Example, details how you can decode one of the channels embedded within a commercial FM broadcast, the RDS or Radio Data System signal.

If you're not familiar, the PySDR.org website represents a whole book about software defined radio and python. It goes into as much or as little detail as you want, to explain how this whole software malarkey works, and takes you by the hand down the path of discovery.

So, armed with a working example, I followed along the bouncing ball and made a working RDS decoder and I think, understood most of it. There's a few interesting wrinkles that I've contacted the author, Dr. Marc Lichtman, about and we'll see what comes of that.

Here's the kicker.

The author, who is also a senior member of the GNU Radio team, started with a GNU Radio flowgraph and reverse engineered what was happening to get to the point of the code that's available in PySDR.org Chapter 18. This is significant because it creates a relationship between the code I have in front of me and the code generated by GNU Radio, which means that when I start with a new flowgraph, not only do I know the steps required, I also know that the outcome is predetermined, as-in, I already know that there's a solution. Having professionally written software for over 40 years, I can tell you that this is not often the case.

I realise that I can search the Internet for an RDS decoder flowgraph, but that's unlikely to get me to a better understanding of what GNU Radio is doing.

Once I've clarified with the author, I'll add the code to my GitHub project, "Fifty Things you can do with a Software Defined Radio", specifically, "Receive road traffic information", since among other things, that's carried by RDS.

As an aside, Rohde and Schwarz have a lovely YouTube video on the topic, "Understanding the Radio Data System", which is giving me a whole set of ideas about things we might attempt with amateur radio repeaters, but that's a story for another time.

Meanwhile, have you considered what other signals exist on the RF spectrum that you might want to decode and how you'd go about this?

I'm Onno VK6FLAB

Foundations of Amateur Radio

How do you make a hole? That's a pretty straightforward kind of question, and by the time this sentence is finished, there's going to be at least as many answers as people who considered it.

I didn't supply any parameters to this hole, so answers could include shovels, collapsing space, fire, a drill, or any number of other interesting approaches. If I narrowed it down to, say, a hole in wood, there'd still be plenty of options. Specifying the type of wood, the diameter and other parameters would further narrow down the selection of methods.

What if I asked you: "How do you decode FM?"

You might wonder if there's more than one way and I can assure you, just like with making a hole, there's plenty of ways to go about achieving this, even if I limit this to software implementations only.

I must confess, when I recently set out to test my Soapy SDR library notions using a GNU Radio flowgraph to listen to FM radio, I searched the documentation, found a beginners tutorial and used the information there to make my first proof of concept FM receiver. I put it on GitHub and went about my business.

After finally managing to hear the decode effort and being less than impressed, I started trying to understand the tutorial flowgraph. When I started looking at what would be needed to decode stereo FM broadcast radio, I discovered that there were several tutorials, examples and videos with slightly, or significantly different solutions to the problem. That's on top of the over a dozen standard FM related blocks supplied within GNU Radio.

I then set about trying to discover the canonical implementation of an FM receiver and came up short. Instead I discovered even more implementations of FM receivers, each subtly different.

You should know that there's a difference between how your local hit radio station does FM and how an amateur radio repeater does FM, let alone the local CB radio channels, satellite telemetry, wireless microphones or even hearing aids, so within the implementation of an FM receiver, there's additional complexity, which explains to some extent the variety of FM related blocks within GNU Radio. I think ultimately it's safe to say that there's an unlimited supply of implementations of an FM receiver within GNU Radio.

It led me to ask, what is the .. for want of a better word .. "right" way and what does that actually mean?

In GNU Radio, you string together blocks that process a signal. If you're familiar with flowcharts, the process is very similar. Unlike flowcharts on a piece of paper, in GNU Radio, or should I say, GNU Radio Companion, the tool you use to actually design flowgraphs, the little blocks represent underlying software and their connections represent how data flows between these bits of software.

In other words, each block represents a series of programming instructions that process data and pass it on. It means that the more blocks you have in your flowgraph, the more instructions are running to process data. The more instructions, the more computing resources required. This is significant because in a complex system like this, we're likely to be doing more than one thing at a time, so preserving resources is important, if only to ensure that there's time available to process the next sample.

As a result, there's a difference between implementing an FM receiver with two blocks, or with ten blocks. You might conclude that two blocks is more efficient, but that might not be true. For example, two blocks processing 2,000 samples per second each, are processing 4,000 samples per second in total. A block that converts the 2,000 samples into 200 samples, followed by nine blocks processing 200 samples per second each, is processing 3,800 samples in total. All things being equal, the ten blocks together are handling less data per second, so overall it's potentially using less resources. I say potentially, because it might be that one of those blocks is using a massive calculation, consuming more resources than all the other blocks put together, ultimately, each block is software, so whatever it's doing is using resources.

So. How would you go about choosing between two implementations or algorithms, which was the "better" one and how is "better" defined?

My first pass at this, is to use standard testing files and using the algorithms under consideration to process them. Run the tests multiple times, keep a record of how long they take and then attempt to measure how much the original input signal differs from the processed output signal. At the moment I have no idea how you might compare signals, other than to invert one and combine them to see if they cancel each other out, which means they're the same, or not, which means that they're different.

For my sins, in trying to think of a way to do this I realised that the way I implement this radio contraption needs to be able to deal with test files and potentially multiple different implementations of a decoder.

It also means that I have some more thinking to do.

Ideally, there needs to be a concept of meta information, like the radio source, the tuned frequency, the bandwidth, gain, and likely more so I can set the parameters once and re-use these across whatever else is part of the flowgraph. It should be possible to use a test file just as simply as a Soapy SDR compatible radio. It should also be possible to hear the audio, or save it to a file, or decode an embedded signal, or all at the same time.

In other words, it needs to be flexible.

Luckily, GNU Radio is really a collection of libraries built precisely for this task.

I'd love to hear your thoughts on the matter.

I'm Onno VK6FLAB

Foundations of Amateur Radio

Still excited from my minor victory in discovering a missing puzzle piece associated with the project I'm working on, I spent the past week introducing my head, if not literally, at least figuratively, to the surface of my desk in a traditional head-desk troubleshooting move that you might be familiar with. I suppose it's an improvement on the "Bear with a Toothache" approach.

In short, the Yak is losing hair .. rapidly.

You might be wondering why I'm telling you about it, since in the land of milk and honey nothing ever goes wrong and all the answers are presented on a silver platter, except when they're not.

Within the amateur radio community, it appears to me that the inclination to fiddle is ingrained and widespread. Given that the hobby is all about experimentation and learning, that's not a bad thing, but there are times when this behaviour can be counterproductive.

Specifically when you're troubleshooting.

Faced with a problem, there are times when a systematic approach is warranted. For some, the first time they come across this phenomenon is during the practical test component of their amateur radio license. Presented with a station, they're asked to determine why it's not working.

The problem might be a power supply that isn't plugged in, or a disconnected antenna, the mode button set to FM, the squelch closed, the RF gain set to zero, generally something simple.

Those inexperienced in the art of troubleshooting are more likely than not inclined to try everything, sometimes all at once, in the hope that one of the changes will magically fix the issue, but in reality, often making the problem worse.

There is a better way.

I'm mentioning this because this skill applies to many aspects of life and in the decades that I've been here, it's not something I was ever taught.

It's funny to think that a quote from nearly a century ago applies to this skill: "When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth."

The salient point being "eliminating all which is impossible".

In other words, you're not finding the problem, you're eliminating all the things that are not the problem. Another way is to think of it as finding all the things that don't need fixing. While that might sound weird, in deeply interdependent systems like an amateur radio station, or a computer, that approach can help you find the root cause of an issue and with it the path to a potential fix.

Of course, this process invites you to examine where the issue might exist. Experience will teach you that you need to start small and grow the net, rather than cast wide and narrow it down. That's not to say that you need to stop paying attention to the bigger picture.

In the example of an inoperative station, you might discover that the lights in the room are out and that this coincides with the radio being off. In other words, trying to discover if the antenna is disconnected makes little sense, since there is no power to the radio.

In my case, I'm balancing my efforts between maintaining an existing system whilst attempting to deploy a new one to replace it. I'm working on several related issues on multiple fronts. Their common theme is audio, though the specifics differ depending on which computer I'm looking at. Then there's the installation and ongoing care and maintenance associated with keeping GNU Radio running.

It's a balancing act because while this is happening, I still need to look for work, respond to email, deal with the regulatory requirements for accounting and tax returns, not to mention the myriad projects I have going on at any given time.

At this point I could go into deep and disturbing detail about the technology issues I'm juggling and I could even justify it by pointing out that a problem shared is a problem halved, but truth be told, I'm not sure you're up for a treatise on the comings and goings of forced system security updates and arising bugs, and just so we're clear, this is not the company who brought us a talking paperclip, it's the one who gave rise to companies called "Orange" and "Lemon".

On the GNU Radio front, there's a snake based installer that happily installs two incompatible libraries for the same application, causing it to fail, and a beer related one that fails to install dependencies. At least I can use 'apt-get' on a real system.

That said, juggling problems and all, I did manage to actually hear an FM station being decoded across the network. It did help that I actually connected the antenna to the radio, and I'm ignoring the audio buffer under-runs, or over-runs, depending on the weather, or some other unknowable variable, but I suspect that's all part of the learning I'm in the middle of.

Hopefully, the hair will grow back soon.

I'm Onno VK6FLAB

Foundations of Amateur Radio

A little while ago I discussed a lovely article by programmer, artist, and game designer "blinry" called "Fifty Things you can do with a Software Defined Radio". This week it occurred to me that I could use their article as a framework to further explore my Bald Yak project. If you're unfamiliar, the Bald Yak project aims to create a modular, bidirectional and distributed signal processing and control system that leverages GNU Radio.

For that to happen, I need a solid understanding of GNU Radio and its ecosystem. While I've been playing with it off and on for a decade or so, I have yet to build anything substantial for the simple reason that there was a puzzle piece missing.

Last week I discovered it .. by accident.

One of the fundamental things I'm attempting to achieve is the creation of a system that doesn't care which radio device you're using. In case you're wondering, I'm doing this because there is a proliferation of device specific software that cannot keep up with the influx of new hardware, doesn't consider the growing use of network connected radios, forced by increased RF noise levels in many communities across the world, not to mention, connecting increasingly expensive computing hardware to lightning rods.

If everything goes to plan, it should be possible to use the project with any radio device. This is easier said than done.

In GNU Radio this complex issue is addressed by having different blocks that represent different devices. You'll find receiver specific source blocks and equivalent sink blocks representing transmitters.

While that's all fine and usable, it means that if I were to publish, say an FM receiver flowgraph, essentially a collection of blocks representing software that implements an FM receiver, I have to decide how I want to deal with the specific device. Do I select an RTL-SDR dongle as the device in my flowgraph and let you figure out how to make it work on the HackRF or the PlutoSDR sitting in your shack, or do I make it completely hardware agnostic, requiring you to wire it all together for your specific situation?

Neither is desirable, or simple.

Added to this is the problem that trying to make this work using a traditional analogue radio would cause more issues, since there isn't a Yaesu FT-857d block, nor is there one for an Icom IC-7400, let alone something from last century.

Someone with some GNU Radio experience might point out that there are source and sink blocks for an audio device, which would allow you to plug one of those radios into a sound card and access the receiver, or transmitter, that way.

While that would work, it requires that the radio is physically connected to a computer that's running GNU Radio. It would also give you all manner of headaches attempting to change frequency in the same way as you could using an RTL-SDR dongle.

There are several ways to get remote radio control working across a network. For example, using 'rigctld' and 'Hamlib', we can change frequency on over 300 analogue radios, but even if you do, you'll discover that getting the audio across the network creates a whole range of new issues, not to mention that GNU Radio doesn't talk to Hamlib compatible radios. This is why many remote radio solutions are implemented as remote desktop sessions to a computer that is physically connected to the radio.

While attempting to solve a completely unrelated challenge last week, I came across 'SoapySDR', described as a vendor and platform neutral SDR support library. Essentially it's a project that allows an application to interact with different devices without needing to support individual radios. This allows an application developer to write their software to support SoapySDR and from then on benefit from its ability to talk to lots of radios in a variety of different ways.

For example, one of the in-built features is called 'SoapyRemote' which allows you to connect to a SoapySDR radio and interact with it across a network. Specifically, you can send and receive, as well as control the radio, essentially bundling together both the audio and control signals.

SoapySDR also includes a tool called 'soapy-audio'. While documentation is sparse, it appears to support Hamlib, which means that you can, at least theoretically, connect a low powered computer, like for example a $5 Raspberry Pi Zero, to your analogue radio and access and control it across the network.

Best part?

It's supported by GNU Radio and many other applications.

I've started creating a repository with the "Fifty Things you can do with a Software Defined Radio", one directory per thing, that will contain the bits needed to run inside GNU Radio and across the network to any SoapySDR compatible hardware.

Now, before you get as excited as I am, there's a few hurdles. I'm not yet sure of the status of soapy-audio, but it looks promising. I have the bits sitting on my computer and I'm working through them. For example, I'm not sure if the current implementation supports transmit. I also have anecdotal evidence that Hamlib support is incomplete, but I don't yet know to what extent. I anticipate some coding in my future. I suspect it will be the equivalent of building a new SoapySDR module, but time will tell.

I can tell you that I'm running an RTL-SDR dongle connected to a low-power computer and I can connect to it across the network with GNU Radio on my workstation and run a proof of concept FM receiver flowgraph. There's no reason for either computer to be in the same room, let alone the same country.

You'll find the project on my VK6FLAB GitHub page. I plan to work my way through the 50 items and discover what I don't know about GNU Radio. Feel free to play along.

I'm Onno VK6FLAB

Foundations of Amateur Radio

How to go about documenting your setup?

Possibly the single most important thing that separates science from "fiddling around" is documentation. Figuring out how to document things is often non-trivial and me telling you that "unless you wrote it down, it didn't happen" only goes so far.

If documentation isn't your thing, what about "I broke something and I don't know how it was before I fiddled" as an incentive instead?

Recently I had cause to explore how to document how my station is configured. To give you a sense, the microphone is connected to a remote-rig, which is connected to a Wi-Fi base station, over Wi-Fi to a Wi-Fi slave, to another remote-rig, to the radio body, to the VHF port, through two coax switches, a run of RG213, to an antenna.

When receiving, it goes from the antenna, to a run of RG213, through two coax switches, to the VHF port, to the radio body, to a remote-rig, to a Wi-Fi slave, to a Wi-Fi base station to a remote-rig, to the remote head, to a set of headphones.

Of course, at this point I've written it down, so, job done .. right?

Well, what about the data connection, the external speaker, the remote head display and other goodies, say nothing of the duplicate devices with similar names. All in all, the FT-857d has something like eleven ports, each remote-rig has ten, so just wording it is a start, but hardly qualifies as documented.

What if we drew a picture instead? At this point you could pull out your crayons and start scribbling on a sheet of butcher's paper and that would be a fine start, but it would be difficult to share with me or anyone else and updating it would be a challenge, let alone versioning it.

As it happens, we're not the first people to have this issue. In the 1980's and 1990's researchers at Bell Labs were trying to figure out how to draw graphs and from that work a language, 'DOT', since everyone is a fan of the "DAG Of Tomorrow", and a series of tools, which today are known as 'Graphviz', made the visualisation of relationships possible without the application of coloured wax on dried cellulose fibre.

In my other, computing job, I had cause to visualise the relationship between a million or so nodes, allowing me to discover a specific node that was directing all traffic, where I could insert my debugging code, but it was only possible thanks to these free and open source tools.

While the DOT language isn't particularly complex, it occurred to me that for someone not conversant with the syntax, we can start even simpler with a CSV text file that shows the relationships between each device and convert the CSV to DOT and in turn to a picture.

For example, I documented the relationship between the radio and the antenna by adding five lines to a CSV file, essentially, FT857d to VHF port to VHF coax switch to VHF grounding switch to RG213 to antenna.

In all, to document everything except power, since I haven't decided how I want to describe it, I used a CSV with 47 lines. On the face of it, that might sound ridiculous, but I can tell you, it shows all the sockets on the FT857d, all the sockets on both remote-rig devices and the relationships between them. With it anyone can duplicate my set-up.

Having previously spent some quality time learning various aspects of the DOT language, I figured I could write a little script to convert CSV files to DOT, but being of the generation of software developers with the attitude, "Why write something if someone else already did?", I discovered that Reinier Post at the Eindhoven University of Technology has a delightful collection of scripts, including one appropriately named 'csv2dot'.

Written in Perl, the only language that according to some looks just as impenetrable before and after encryption, the tool works as advertised and makes a DOT file that you can then visualise using Graphviz.

Of course there's Python scripts lying around that claim to do the same, but I wasn't keen to install the kitchen sink just to try them. Instead I made a quick little Docker file that you can find on my vk6flab GitHub repository that will walk you through this, complete with my example, so you have a starting point.

Now, I used this here to describe my station, well, one part of it, but it can easily extend to document your entire station, and because we're talking about text files that contain the information, anyone with a copy of a text editor can update the file when things change, since that's where the real magic happens.

So, what are you waiting for, documentation?

I'm Onno VK6FLAB