AXN10 Woes

I bought a Cambridge Audio AX system for my office because it was cheap and it let me play records too. It’s really not too bad but I have my suspicions over the AXN10 Network Streamer. I’ve noticed it has a tendency to stop playing after a while. It plays fine for a good four to six months and then it’s like it runs out of buffer space or something and can only play music in five second chunks followed by a five second pauses. It’s peculiar. It’s also done it three times now so it’s now something that needs investigating.

My current setup uses my music stores on a WD MyCloud Mirror (that also needs fixing) that is streamed out via AssetUPnP running on an Ubuntu 22.04.5 LTS box to my CXNv2 and my AXN10. I originally used Twonky but Twonky is now, effectively, a dead platform. It doesn’t and never will support the Disk Number fields so multi-disk albums are a problem with Twonky.

I was hoping that I could use AssetUPnP to log the music and the duration played but I’m not convinced it’s logging is that capable. It seemed to log connections from the TV but not from the CA AXN which is a bit of a problem when I’m interested in what the AXN is doing.

The AXN itself doesn’t seem to do any logging either but, obviously, the Stream Magic app connects over APIs to the device so I was wondering whether there was a hidden interface there that I could poke. I’m really not the best when it comes to reverse engineering interfaces so I did something really dumb and just put in the IP address of the AXN into a browser followed by /debug. Surprisingly, this returned a “Remote debug not enabled” message which immediately made me think that there must be a way to enable remote debug. Time to go digging…

Ecosia wasn’t really of much help here but Google, especially Googles AI bot was most helpful. Initially it pointed me to a very nice project on GitHub (https://github.com/noahhusby/aiostreammagic)

Now, I’m no python developer (I’m no developer at all, really) but by explaining what I was trying to achieve Google came up with the following program:

import asyncio
import csv
import os
import sys
from datetime import datetime
from aiostreammagic import StreamMagicClient

# CONFIGURATION
AXN10_IP = "IP GOES HERE"  # Replace with your exact AXN10 IP
LOG_FILE = "/usr/local/bin/axn10-tracker/streamer_playback_log.csv"

class PlaybackTracker:
    def __init__(self):
        self.current_track = None
        self.current_artist = None
        self.play_start_time = None
        self.last_state = None

    def log_to_csv(self, start_time, stop_time, artist, track, duration_seconds):
        """Appends a completed playback record to the CSV file."""
        file_exists = os.path.isfile(LOG_FILE)
        
        try:
            with open(LOG_FILE, mode='a', newline='', encoding='utf-8') as f:
                writer = csv.writer(f)
                if not file_exists:
                    writer.writerow(["Start Time", "Stop Time", "Artist", "Track", "Played Duration (Sec)", "Played Duration (Min)"])
                
                duration_minutes = round(duration_seconds / 60, 2)
                writer.writerow([
                    start_time.strftime("%Y-%m-%d %H:%M:%S"),
                    stop_time.strftime("%Y-%m-%d %H:%M:%S"),
                    artist or "Unknown Artist",
                    track or "Unknown Track",
                    round(duration_seconds, 1),
                    duration_minutes
                ])
            print(f"Logged to CSV: '{track}' by {artist} - {duration_minutes} mins.", flush=True)
        except Exception as e:
            print(f"Failed writing to CSV: {e}", file=sys.stderr, flush=True)

    async def on_state_change(self, client: StreamMagicClient, *args):
        """Official callback engine triggered immediately on AXN10 updates."""
        # Grab the core play state object safely
        ps_obj = client.play_state
        if ps_obj:
            # Extract simple state string (e.g., 'play', 'pause', 'stop', 'idle')
            play_state = getattr(ps_obj, 'state', 'stopped').lower()
            
            # Extract nested metadata objects
            metadata = getattr(ps_obj, 'metadata', None)
            track_title = getattr(metadata, 'title', None) if metadata else None
            artist_name = getattr(metadata, 'artist', None) if metadata else None
        else:
            play_state = "stopped"
            track_title = None
            artist_name = None

        # Clean diagnostic line showing exactly what song has been processed
        print(f"[TRACKER STATUS] State: {play_state.upper()} | Title: '{track_title}' by {artist_name}", flush=True)

        now = datetime.now()

        # CASE 1: Streamer just entered a play cycle
        if play_state == "play" and self.last_state != "play":
            self.play_start_time = now
            self.current_track = track_title
            self.current_artist = artist_name
            self.last_state = "play"
            print(f"Playback started: '{track_title}' by {artist_name}", flush=True)

        # CASE 2: Tracks shifted during active streaming
        elif play_state == "play" and self.current_track != track_title:
            if self.play_start_time:
                duration = (now - self.play_start_time).total_seconds()
                if duration > 2:
                    self.log_to_csv(self.play_start_time, now, self.current_artist, self.current_track, duration)
            
            self.play_start_time = now
            self.current_track = track_title
            self.current_artist = artist_name
            print(f"Track changed: '{track_title}' by {artist_name}", flush=True)

        # CASE 3: Playback stopped, paused, or returned to idle
        elif play_state in ["pause", "stop", "idle"] and self.last_state == "play":
            if self.play_start_time:
                duration = (now - self.play_start_time).total_seconds()
                if duration > 2:
                    self.log_to_csv(self.play_start_time, now, self.current_artist, self.current_track, duration)
            self.play_start_time = None
            self.current_track = None
            self.last_state = play_state
            print(f"Playback stopped ({play_state}).", flush=True)

async def main():
    tracker = PlaybackTracker()
    print(f"Initializing event pipeline for AXN10 at {AXN10_IP}...", flush=True)
    
    while True:
        try:
            client = StreamMagicClient(AXN10_IP)
            
            # 1. Bind our handler logic block
            await client.register_state_update_callbacks(tracker.on_state_change)
            
            # 2. ACTUALLY EXECUTE THE NETWORK HANDSHAKE CONNECTION
            await client.connect()
            
            print("Connected! Active socket listening for streaming metrics...", flush=True)
            
            # Keep the context block alive indefinitely
            while True:
                await asyncio.sleep(3600)
                
        except Exception as e:
            print(f"Network drop encountered: {e}. Reconnecting in 15 seconds...", file=sys.stderr, flush=True)
            await asyncio.sleep(15)

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("\nShutdown signal captured.")

This logs every event to a CSV and lets me keep track of everything that has been played and for how long. It also came up with a companion program to summarise the data for me. I probably could have done that bit myself but I was more interested in having something work that hacking around trying to make things work.

import csv
from datetime import datetime
from collections import defaultdict

LOG_FILE = "/usr/local/bin/axn10-tracker/streamer_playback_log.csv"

def analyze_logs():
    total_seconds = 0
    track_counts = defaultdict(int)
    track_durations = defaultdict(float)
    
    try:
        with open(LOG_FILE, mode='r', encoding='utf-8') as f:
            reader = csv.DictReader(f)
            
            for row in reader:
                # Calculate overall cumulative time
                seconds = float(row["Played Duration (Sec)"])
                total_seconds += seconds
                
                # Group data by track identifier
                track_identifier = f"'{row['Track']}' by {row['Artist']}"
                track_counts[track_identifier] += 1
                track_durations[track_identifier] += seconds
                
    except FileNotFoundError:
        print(f"Error: Could not find '{LOG_FILE}'. Make sure the logger script has run and recorded at least one song.")
        return

    # Convert total seconds into hours and minutes
    total_minutes = total_seconds / 60
    total_hours = total_minutes / 60
    
    print("=" * 50)
    print("      CAMBRIDGE AUDIO AXN10 PLAYBACK REPORT      ")
    print("=" * 50)
    print(f"Total Listening Time: {total_hours:.2f} Hours ({int(total_minutes)} Minutes)")
    print("-" * 50)
    
    # Sort and find the tracks listened to for the longest total duration
    print("Top 5 Most Listened-To Tracks (By Actual Duration):")
    sorted_by_duration = sorted(track_durations.items(), key=lambda x: x[1], reverse=True)
    
    for i, (track, seconds) in enumerate(sorted_by_duration[:5], 1):
        mins = seconds / 60
        print(f" {i}. {track} — Listened for {mins:.1f} mins total")

if __name__ == "__main__":
    analyze_logs()

The next thing we worked on was making it into a service so that it would always run. The last thing I needed was to forget to log a week or month’s worth of data and complete stuff my numbers up. Adding it as a service was pretty trivial.

First create the logger service:

sudo vi /etc/systemd/system/axn10-logger.service

   [Unit]
   Description=Cambridge Audio AXN10 Playback Logger
   After=network-online.target
   Wants=network-online.target

   [Service]
   Type=simple
   User=YOUR_UBUNTU_USERNAME
   WorkingDirectory=/usr/local/bin/axn10-tracker
   ExecStart=/usr/local/bin/axn10-tracker/venv/bin/python3 /usr/local/bin/axn10-tracker/axn10_logger.py
   Restart=always
   RestartSec=10
   StandardOutput=journal
   StandardError=journal

   [Install]
   WantedBy=multi-user.target

then enable and run the service:

sudo systemctl daemon-reload
sudo systemctl enable axn10-logger.service
sudo systemctl start axn10-logger.service

It actually took a few iterations to get the logging code to properly run as a service and to log the data correctly but Google helped me achieve something I could not have done on my own.

So now, every time I play something on my AXN it gets logged to a file and over time it will show me, not only the total play time between failures, it will also show me what artists / albums / songs I’m listening to the most. That’s pretty cool.

Ubuntu 24.04 on a Terra-Master T12-423

My reasons for putting Ubuntu are explained here: Confession Time and I’m using Ubuntu because that’s what Terra-Master use themselves so it should work just fine and it’s the distribution I’m slightly more familiar with at the moment. Debian would be my alternative choice.

First things first. Open up the case and remove the small SanDisk USB stick. This contains the TOS boot loader and we don’t want that anymore. Go and put it in a marked envelope in a drawer somewhere just in case you ever want to put it back.

Step 2 – not for the faint of heart. Update the BIOS. The full thread is here: https://forum.terra-master.com/en/viewtopic.php?t=4238 and with that done the rest is pretty straightforward.

Next we need to change the boot order. With the TOS USB stick removed that’s an awful lot easier. For some reason, for me at least, the TOS USB stick always took priority no matter what I did so with that out of the way it’s a lot easier to do things now.

I actually burned Ubuntu to DVD rather than a flash drive simply because I don’t think the flash drives I have are very reliable. They are incredibly slow for some reason too. So, I opted for DVD.

Slap the DVD in and run through the basic setup until the drive configuration page presents itself. Here choose the custom layout option. I have two Seagate IronWolf disks as my boot disks so I wanted a mirrored raid boot setup. Not having done that before I found a very easy to follow video (and I really hate having to follow videos) here:

Installing Ubuntu 24.04 With RAID

and a really useful page here:

Ubuntu 24.04 with software RAID1 and UEFI

The only changes to the process I made was to include in my four 10TB raid disks into the process. That means that at the end of the video I have a filesystem that looks something like:

md0 RAID1 /boot
md1 RAID1 SWAP
md2 RAID1 /root
md3 RAID5 /storage

It also means that as the storage is built at installation time I can just sit back and let the RAID configure itself for a few hours.

Because I always choose a minimised installation I need to add a few things in to make the environment usable so this is what I do:

sudo apt update
sudo apt install htop
sudo apt install avahi-daemon
sudo apt install vim
sudo apt install cron
sudo apt upgrade

Once cron and vim are installed I set up my lazy mdstat rebuild page.

I have a little web server with a basic html page of:

<!DOCTYPE html>
<html>
<head>
<title>
RAID Rebuild Status
</title>
<meta http-equiv="refresh" content="60" >
</head>
<body>
<embed src="raid.txt" width="100%" height="1000">
</body>
</html>

It’s really not much of a page. Then on the T12 I add in a cron job:

* * * * * cat /proc/mdstat | ssh ricardo@server.local 'cat - > /home/ricardo/docker/dumkid/html/raid.txt'

This means that I can point my phone / laptop / work computer to my little webpage and see how the rebuild is going without having to log in to the T12 itself.

Confession Time

My WD PR4100 has been pretty stable over the years but, despite having 40TB of disks in there, I’ve run out of space. I *could* have tried to iSCSI some storage but I really just wanted something bigger. My primary requirement is Plex but I’d also quite like to put Twonky on there too.

I quite like the look of the Terra-Master T12-423 because it has lots of expansion space, would easily let me do RAID 6 and it looks like nice too. However, reading around on various forums there was a general consensus of opinion that the OS was pretty poor and it would be better to run UnRaid or TrueNAS on the thing. Well, I’ve always been a sucker for the underdog and quite like a challenge so that’s the device I went for.

What can I say? Well, it was packaged very nicely and in a way that appeals to my aesthetics. The tin itself is very nice too. No complaints there. They even give you a nice little screwdriver for the disks and the case. Almost everybody has a screwdriver so it’s really not necessary but it’s a really nice touch that means you have one immediately to hand.

The rest of it… well, let’s just say the comments on the forums weren’t even close. With the PR4100 you just have to slap some disks in and it’s ready to go. There’s some small configuration that can be done to personalise the device but, out of the box, it just works. The T12-423 is not like there. There’s a small USB boot loader that prompts for a disk but it doesn’t tell you what for. After giving it a disk for storage it installed the OS. That’s not quite what I wanted. Fine. Slap a couple of more disks in there and see what it does. After a good twelve hours of synchronising the test disks are ready. Great! Let’s play.

First off, basic public shared access. I want this to be a media server so I just want to be able to drop files into the shares without having to authenticate first. It did not work. Guest user enabled, read / write permission given to both the user and the group. Nothing but NTLM authentication issues. Thing is, it’s not even possible to hack around with the Samba config because the TOS constantly reverts it to what it thinks it should be.

While trying to hack around with the Samba config I found that it isn’t possible to copy files off the device using SCP because there appears to be some security program running in the background that just kills the process. It’s not even possible to use SFTP because that just segfaults. SFTP has also been configured to run on a non-standard port for “security” reasons and it’s not possible to change back.

The security limitations and alerts are so annoying. It’s not possible to have a user with a name containing “super” or “admin”. This admin account is the only account that can SSH into the box which means that it could effectively become a shared account which is even worse from a security point of view. The security alerting is just dumb. It raises alerts for things like not isolating the box on the network, using the non-standard default ports for SSH, HTTP, HTTPS and such like. It allows the easy use of both Telnet and FTP though which is just insane in today’s world.

Notifications and email are tied together so every time you change a notification setting you need to give it the email password. That makes no sense, they are distinct items and should be treated as such.

Adding disks into the array is really painful. Each disk has to be added one at a time and the array has to completely rebuild before the next one can be added in. When you’re building the initial array that’s just painful. It took a week to build an array using four 10TB disks.

Support is not very good either. When posting on the forum about the Samba issue on the public share I described the issue and everything I’d done up to that point only to get a response telling me I needed to enable the Guest user. Once I’d found out how to enable logging and posted the relevant log lines support stopped responding.

The support files the system generates are too big, or contain too many folders, to be sent through most mail servers. There is no FTP server when the files can be dropped either. Apparently it is possible to remove some of the folders from the file but if they’re not important why collect them anyway? The other issue is that the support team are based in Hong Kong so, given the time difference, it usually takes a day to get a response. If you get one at all.

All-in-all I’ve had the hardware for a week now and still not managed to move any of my media over from the PR4100. TOS really is poor and some very, very strange design choices have been made that make it really hard to work with. Support is practically non-existent and the software just isn’t fit for purpose. Still, the good thing is that it is generic x86 hardware so it’s not too hard to install something else…

Raid Rebuild

Terra-Master seems to use mdadm:

TNAS:~# ps -ef | grep [m]dadm
davros       547       1  0 Jan12 ?        00:00:00 /sbin/mdadm --monitor --scan --daemonise --syslog --delay=10 --program=/etc/init.d/nas/mdadm-monitor

so, it looks like the best way to follow the rebuild is:

TNAS:~# watch cat /proc/mdstat 

which will generate an output like:

Every 2.0s: cat /proc/mdstat                                                                                                                                                                                   TNAS: Mon Jan 13 09:19:44 2025

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md1 : active raid1 sdd4[1] sda4[0]
9755199616 blocks super 1.2 [2/1] [U_]
[===>.................] recovery = 18.9% (1852244544/9755199616) finish=667.2min speed=197414K/sec

md0 : active raid1 sdc4[1] sdb4[0]
3895824636 blocks super 1.2 [2/2] [UU]

md8 : active raid1 sdc3[1] sdb3[0]
1997824 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md9 : active raid1 sdc2[2] sdb2[0]
7995392 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>